arXiv AI By Carson Rodrigues

A Contradiction Judge Counted Reported Teammate Errors as Hallucinations: An Assertion-Aware Re-Score of One Multi-Agent LLM Harness

Read the original on arXiv AI →

The paper revises earlier claims about hallucination rates in multi‑agent LLM systems, noting that a judge that flags contradictions can mistakenly label correct reports of teammate errors as hallucinations. Using a pre‑registered analysis plan, the authors re‑score outputs from a harness with Llama and Claude agents, finding that the broadcast effect on hallucination rates is negligible and that most hallucinations stem from teammates’ wrong values. The study also reports that the effect varies across judge‑prompt combinations and that the data are limited to a single harness and 13 hand‑written scenarios.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 26

Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.

By Ancil Crayton
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv AI
Oct 2

The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating

The paper demonstrates that reading a large language model (LLM) judge’s verdict from the logits of its first generated token—an approach used in constrained decoding and likelihood‑scoring evaluation—introduces a significant distortion in position bias. Because judges do not always start with a verdict token (12–49% of cases for Qwen3 judges and <3% for Llama‑3.1‑8B and Phi‑3.5‑mini), this readout often returns the first response rather than a true judgment, inflating position bias by up to 42 points while barely affecting judge accuracy. The authors recommend reporting the frequency with which a judge leads with a verdict token to provide a more accurate assessment of position bias.

By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli