arXiv AI
Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
1d ago
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
AGORA is an item‑response‑theory aggregator that improves the reliability of large language model judges by learning each judge’s sensitivity and rating thresholds from all items jointly, without needing human labels. It places every item on a latent quality scale, allowing judge reliability to be inferred from the data itself. Across three benchmarks, AGORA matches or surpasses the Dawid‑Skene method, especially when evaluation sets are small or judge pools are uneven.
arXiv AI
Sep 28
Accounting for Bias Enables Sustainable LLM Evaluation
The paper argues that the current LLM-as-a-judge evaluation method, which compensates for systematic measurement bias by increasing the number of comparisons, is statistically unsound and computationally wasteful. It identifies that treating LLM judges as neutral ignores documented biases such as position bias, verbosity bias, judge severity, and self‑enhancement. The authors propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, enabling reliable rankings with far fewer comparisons and negligible additional compute.
arXiv AI
Jul 10
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
arXiv:2607. 08535v1 Announce Type: cross Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.
arXiv AI
Sep 23
Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
arXiv:2609.22512v1 Announce Type: new Abstract: Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In p...
arXiv Machine Learning
Jun 5
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
arXiv Computation and Language
1d ago
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
arXiv:2603.01865v4 Announce Type: replace Abstract: LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be aver...