arXiv AI August 10, 2026 By Yang Shu

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

arXiv:2608. 06940v1 Announce Type: new Abstract: LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 1

RoPoLL: Robust Panel of LLM Judges

arXiv:2606. 30931v1 Announce Type: new Abstract: The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood.

By Anish Acharya, Kris W Pan, Brian Verkhovsky

llms benchmarks safety

arXiv Machine Learning

Jul 21

When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

arXiv:2607. 02104v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

llms safety

arXiv Machine Learning

1d ago

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

arXiv:2608. 15980v1 Announce Type: cross Abstract: Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.

By Anik Jha

llms benchmarks safety

arXiv AI

Aug 11

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines

arXiv:2608. 07813v1 Announce Type: new Abstract: An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships.

By Yiyao Zhang, Diksha Goel, Hussain Ahmad, Shixun Huang, Jun Shen

llms

arXiv AI

Jul 10

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.

By Kaihua Ding

llms safety

arXiv AI

Jun 15

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

arXiv:2606. 13685v1 Announce Type: cross Abstract: LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized.

By Abel Yagubyan

llms safety

Related stories

RoPoLL: Robust Panel of LLM Judges

When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines

When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation