arXiv AI By Junhyung Ahn

AGORA: Reliability-Aware Aggregation of LLM Judges on Small Evaluation Sets

Read the original on arXiv AI →

AGORA (Aggregation Of Reliability-weighted Assessments) is an item‑response‑theory aggregator designed for LLM judges on small evaluation sets. It models each judge with a sensitivity parameter and ordered rating thresholds, learning judge reliability jointly from all items without relying on human labels. Across three benchmarks, AGORA matches or surpasses Dawid‑Skene on large sets and performs better as the set size shrinks, remaining robust even with uneven judge pools and providing a label‑free statistic to gauge when trust weighting is beneficial.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Nov 5

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

AGORA is an item‑response‑theory aggregator that improves the reliability of large language model judges by learning each judge’s sensitivity and rating thresholds from all items jointly, without needing human labels. It places every item on a latent quality scale, allowing judge reliability to be inferred from the data itself. Across three benchmarks, AGORA matches or surpasses the Dawid‑Skene method, especially when evaluation sets are small or judge pools are uneven.

By Junhyung Ahn
arXiv Computation and Language
Sep 25

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

The study investigates whether Jev, a typed classifier that outputs probabilities over allowed answers without generating text, can replace large language model (LLM) rubric judges. Across nine panels from seven benchmarks, Jev’s accuracy differed significantly from LLM judges in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons were inconclusive. In terms of cost and speed, Jev was 29 to 325 times cheaper and 30 to 220 times faster than the flash‑tier LLM judges, and a cascade approach that defers uncertain Jev verdicts to an LLM yielded only modest gains. whyItMatters":"The findings suggest that a lightweight classifier like Jev can serve as an efficient first‑stage evaluator, potentially reducing the reliance on expensive and slow LLM judges in automated grading pipelines."

By Delip Rao, Chris Callison-Burch
arXiv Machine Learning
Sep 14

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.

By Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan, Sulaiman Vesal
arXiv Computation and Language
Nov 5

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

The paper evaluates 21 LLM-as-a-Judge models from nine providers across MT-Bench, JudgeBench, and RewardBench, using agreement, consistency, and bias audit protocols over 118 runs and about 541,000 judgments. It finds that exact-match agreement overstates performance, with Cohen’s kappa lower by 33–41 percentage points, judge rankings shift up to 14 positions, and high test–retest reliability coexists with significant position bias in some production judges. Verbosity bias is minimal across the cohort. The authors propose a Minimum Viable Validation Protocol based on these findings.

By Justin D. Norman, Michael U. Rivera, D. Alex Hughes