arXiv AI
LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 28
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
AgentJudgeBench is a new benchmark that evaluates the reliability of large language model (LLM) judges on agentic tool‑calling tasks involving workflow directed acyclic graphs (DAGs). It contains 3,808 instances across six DAG topologies and three difficulty tiers, tested with five generators (3B–70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under both paired‑with‑and‑without‑ground‑truth conditions. The study finds that judge alignment degrades with task difficulty, ground‑truth exposure can sometimes hurt alignment, and structured evaluation rubrics provide modest improvements, revealing a structural ceiling that model capacity alone cannot surpass.
arXiv AI
Oct 1
Hard-Gate Candidacy in a Deployed Validator Suite
The paper evaluates hard‑gate candidacy for validators in a deployed generative‑agent system by measuring how well each validator’s firing separates successful from failed builds. Across 13 validators and thousands of builds, only a few checks show statistically significant separation, while many fail to distinguish or never fire. The study highlights that skipped checks are recorded as passes, limiting detectable failure rates and underscoring the need for clearer evaluation records.
arXiv AI
Aug 11
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
arXiv:2608. 07813v1 Announce Type: new Abstract: An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships.
arXiv AI
Jun 15
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
arXiv:2606. 13685v1 Announce Type: cross Abstract: LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized.
arXiv AI
Oct 16
Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation
The paper introduces DIB-Bench, a benchmark that exposes deployment‑integrity blindness in large language model (LLM) evaluation by testing 7,560 scored examples across six tamper types and a range of evaluators from regex scorers to GPT‑5‑nano. It shows that stronger evaluators are more likely to accept tampered outputs—mean acceptance rises from 88.6% for rule‑based scorers to 94.1% for GPT‑5‑nano—and that certain trigger‑conditional tampering can produce harmful outputs in 48–100% of cases, increasing with model capability. The study also demonstrates that output‑only evaluation cannot detect tampering with zero output shift, while a manifest‑hash identity channel can detect it with zero samples, and proposes a graded deployment‑identity descriptor for evaluation reports.
arXiv Machine Learning
Oct 3
VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence
VERITYGATE is a four‑gate framework that checks whether LLM‑generated narrations adhere to a fixed schema of declared evidence IDs, entities, numbers, and claim types, rather than verifying every fact in the prose. In experiments with GPT‑4o‑mini, Llama‑3.3‑70B, and Claude Sonnet 4.6 on 900 grounded‑ungrounded pairs, the framework identified high failure rates (up to 80.3% for mini claims) and demonstrated that a single repair pass can improve claim survival rates. The authors also provide code, data, and preliminary human studies to validate the rules and highlight gaps between schema compliance and correct prose. "whyItMatters":"The framework offers a systematic way to evaluate and improve the faithfulness of LLM explanations to structured evidence, revealing significant failure rates and the impact of repair strategies."