arXiv AI By Ian Arawjo

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv Machine Learning
Sep 14

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.

By Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan, Sulaiman Vesal
arXiv Computation and Language
Sep 11

Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.

By Yu-Chung Hsiao
arXiv Computation and Language
Oct 22

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

SANE is an agent‑based framework that automatically generates scorer‑specific perturbation recipes and expected score relations for large‑language‑model judges, using behavioral requirements encoded as reusable agent instructions. It validates each recipe against the scorer’s intent and the instantiated test case, producing a growing inventory of aligned recipes as more scorers are added. In experiments across five domains, SANE yields 3.7× more validated test families than the Judge Reliability Harness, detects deficiencies in all domains—including professionalism and toxicity—and achieves strong human alignment in judge ranking on faithfulness (Spearman ρ = 0.852).

By Ethan Ludwick, Yu-Chung Hsiao