arXiv AI

Cherry-pick Override: LLM Judges Under-use the Non-Directional Verdicts Their Contract Authorizes

arXiv AI
3d ago

A Contradiction Judge Counted Reported Teammate Errors as Hallucinations: An Assertion-Aware Re-Score of One Multi-Agent LLM Harness

The paper revises earlier claims about hallucination rates in multi‑agent LLM systems, noting that a judge that flags contradictions can mistakenly label correct reports of teammate errors as hallucinations. Using a pre‑registered analysis plan, the authors re‑score outputs from a harness with Llama and Claude agents, finding that the broadcast effect on hallucination rates is negligible and that most hallucinations stem from teammates’ wrong values. The study also reports that the effect varies across judge‑prompt combinations and that the data are limited to a single harness and 13 hand‑written scenarios.

By Carson Rodrigues
arXiv AI
Sep 2

Commit-first LLM judging inherits the judge's own errors

The paper investigates whether widely used evaluation frameworks for large language models (LLMs) implement a defense called commit‑first judging, which requires a judge to solve a task itself before accepting a candidate answer. Across 24 configurations in eight popular frameworks, none use the full commit‑first method; nine use a weaker variant that is ineffective. In controlled experiments, the weaker variant allowed systems to game the judge, while the full commit‑first approach eliminated this vulnerability but sometimes worsened evaluation when the judge’s own answer was incorrect.

By Idil Gozel
arXiv AI
Oct 12

Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans

The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.

By Sangdo Han, Moonsoo Kim
arXiv Computation and Language
Oct 16

Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.

By Alden Do Rosario, Kirollos Atef
arXiv AI
Aug 24

JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification

JuryProbe is an empirical diagnostic tool designed to assess consensus risk in panels of reference‑free large language model judges used for factuality verification. It estimates risk by measuring false‑negative correlations and false‑consensus lift from a labeled calibration probe, and routes high‑risk majority decisions to judges with trusted references. The approach was validated on FEVER corruptions, showing that flagged decisions can be grounded without additional reference acquisition in most cases, while reducing false accepts by about 0.4% and avoiding 28% of reference acquisitions.

By Tianxin Zhou, Ruixi Lin
arXiv AI
Oct 27

Trainable Boundary-Keeping Where No Formal Answer Exists: A Controlled Knock-In Study of Reward Training in Evidence-Sufficiency Judgment

The study evaluates whether reward‑trained language models can learn to keep evidence‑sufficiency boundaries without a formal correctness signal. Using a controlled knock‑in design, a boundary‑keeping recipe (SFT followed by GRPO) dramatically increased spontaneous correct refusals from 25 % to 100 %, while a scrambled‑pairing placebo collapsed performance. The results show that final boundary‑keeping is trainable without an inference‑time verifier, though active numeric checking remains negligible.

By YaJie Yin
arXiv AI
Aug 26

More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.

By Yuchen Han, Cheng Yan, Wuyang Zhang