arXiv AI By Haoran Xu

Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
3d ago

A Contradiction Judge Counted Reported Teammate Errors as Hallucinations: An Assertion-Aware Re-Score of One Multi-Agent LLM Harness

The paper revises earlier claims about hallucination rates in multi‑agent LLM systems, noting that a judge that flags contradictions can mistakenly label correct reports of teammate errors as hallucinations. Using a pre‑registered analysis plan, the authors re‑score outputs from a harness with Llama and Claude agents, finding that the broadcast effect on hallucination rates is negligible and that most hallucinations stem from teammates’ wrong values. The study also reports that the effect varies across judge‑prompt combinations and that the data are limited to a single harness and 13 hand‑written scenarios.

By Carson Rodrigues
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv AI
Oct 12

Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans

The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.

By Sangdo Han, Moonsoo Kim