arXiv AI

Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?

arXiv AI
Sep 28

When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

The paper investigates when a language‑model judge can truly ground its verdicts in code correctness. It shows that current multi‑agent verification methods rely on evidence that is both independent of the answer and distinct between candidates—conditions that fail in code judging. By analyzing two label‑free measurements from the judge’s logs, the authors demonstrate that gating on one measurement allows the system to decline uncertain comparisons, improving accuracy from 20.7% to 36.9% while still answering half of all cases.

By Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
arXiv Computation and Language
Oct 9

MiniVer-V: Identifying Minimal Sufficient Evidence for Short Video Verification

MiniVer‑V is a new benchmark for short‑video fact‑checking that contains 195 videos, each annotated with a supported, refuted, or insufficient verdict and 5,510 multimodal evidence units (visual keyframes, speech transcripts, and web‑retrieved sources). The authors propose a two‑layer verification framework that first checks claim‑video consistency using internal evidence, then determines the factual verdict with external corroboration, and employs a sufficiency‑driven greedy search to assemble only the minimal evidence needed for a confident decision. Using Claude Sonnet 4, the method achieves a Macro‑F1 of 0.510 with an average of 4.5 evidence units—about 16 % of the full set—while matching the full‑evidence baseline and improving detection of insufficient cases.

By Leran Chen, Lingnan Kong, Zile Cai
arXiv AI
Oct 5

Sentence-Level Context Sensitivity as a Training-Free Detector of Unsupported Content, Evaluated Against Trained Verifiers

The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.

By Mohamed Aly Bouke
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv Computation and Language
Nov 5

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

The paper evaluates 21 LLM-as-a-Judge models from nine providers across MT-Bench, JudgeBench, and RewardBench, using agreement, consistency, and bias audit protocols over 118 runs and about 541,000 judgments. It finds that exact-match agreement overstates performance, with Cohen’s kappa lower by 33–41 percentage points, judge rankings shift up to 14 positions, and high test–retest reliability coexists with significant position bias in some production judges. Verbosity bias is minimal across the cohort. The authors propose a Minimum Viable Validation Protocol based on these findings.

By Justin D. Norman, Michael U. Rivera, D. Alex Hughes
arXiv AI
Oct 2

The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating

The paper demonstrates that reading a large language model (LLM) judge’s verdict from the logits of its first generated token—an approach used in constrained decoding and likelihood‑scoring evaluation—introduces a significant distortion in position bias. Because judges do not always start with a verdict token (12–49% of cases for Qwen3 judges and <3% for Llama‑3.1‑8B and Phi‑3.5‑mini), this readout often returns the first response rather than a true judgment, inflating position bias by up to 42 points while barely affecting judge accuracy. The authors recommend reporting the frequency with which a judge leads with a verdict token to provide a more accurate assessment of position bias.

By Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
arXiv Computation and Language
Sep 24

What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs

The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.

By Han Chen, Yingrui Li
arXiv Computation and Language
Sep 21

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

JudgeSense is a benchmark comprising 880 items from human‑labelled corpora, each presented under two differently worded instructions that ask the same question. The study evaluates 25 judges from six providers across four tasks, measuring how rewording affects agreement with the judge’s own verdicts. Results show that rewording reduces agreement on all tasks, with significant effects on two, and that stability varies across tasks and is not predicted by parameter count.

By Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang