arXiv AI
Jul 23
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
arXiv:2604. 06996v2 Announce Type: replace-cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.
Quantifying and Mitigating Self-Preference Bias of LLM Judges
arXiv:2604. 22891v4 Announce Type: replace-cross Abstract: LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on.
Related stories
arXiv Machine Learning
Sep 14
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.
arXiv Machine Learning
Sep 14
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
GAUGE is a new offline protocol that evaluates whether the common practice of using an LLM-as-a-judge to rank task‑oriented agents actually aligns with a verifiable reward. Across 25 agents from six providers on two benchmarks, GAUGE finds that user satisfaction scores are largely uncorrelated with task success, and that the judge’s ranking loses precision when agents are closely matched in performance. The study highlights a gap between ranking validity and construct validity in current evaluation practices.
arXiv Computation and Language
Aug 27
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
The study investigates how prior scores influence large language model (LLM) judgments in the LLM-as-a-Judge paradigm. By testing three prompt conditions—no metadata, revision framing, and anchored metadata containing prior scores—the authors find that prior scores systematically bias evaluations, shifting ratings toward those scores across 192,000 attempts. The bias also affects categorical decisions, blocking 48% of error corrections and flipping 10.18% of correct judgments, and is not mitigated by Chain-of-Thought or a warning, underscoring the need for careful context engineering.
arXiv Machine Learning
Jun 15
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.
arXiv AI
Jun 24
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
arXiv:2509. 03647v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models.
arXiv Computation and Language
Sep 16
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
The paper introduces JudgeBiasBench, a benchmark that systematically quantifies judgment biases in large language model (LLM)-based judges across four dimensions and 12 bias types. It evaluates both generative and discriminative judges, revealing significant bias patterns that undermine reliability. The authors propose bias-aware training—reinforcement learning for generative judges and contrastive learning for discriminative judges—to reduce these biases while maintaining evaluation performance.
arXiv AI
Sep 1
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
The paper investigates the issue of self‑bias when large language models (LLMs) generate and evaluate their own benchmarks. Using machine translation as a testbed, it shows that LLMs as both test‑set creators and evaluators produce model‑specific, homogeneous outputs that inflate their own scores, even when diversity controls are applied. The bias is strong enough that each model ranks itself first, overriding peer consensus, and the phenomenon also appears in open‑ended generation tasks.
arXiv AI
4d ago
JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
JudgeProfile is a framework that analyzes the subjectivity of large language model (LLM) judges by separating evaluation into perception—how judges compare responses on attributes such as clarity, correctness, and detail—and prioritization—how much each attribute influences the final decision. Using the curated SubjectiveSet dataset of 50,013 response pairs evaluated by 21 judges across 87 attributes, the study finds that judges often agree on attribute judgments even when their overall choices differ. By estimating and adjusting attribute weights, the authors improve agreement with reference labels from 66.48% to 71.97%, outperforming fine‑tuning and rubric prompting.
arXiv AI
Jun 3
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks
arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.
arXiv AI
6d ago
Accounting for Bias Enables Sustainable LLM Evaluation
The paper argues that the current LLM-as-a-judge evaluation method, which compensates for systematic measurement bias by increasing the number of comparisons, is statistically unsound and computationally wasteful. It identifies that treating LLM judges as neutral ignores documented biases such as position bias, verbosity bias, judge severity, and self‑enhancement. The authors propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, enabling reliable rankings with far fewer comparisons and negligible additional compute.
arXiv Machine Learning
Aug 18
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data
arXiv:2410. 13341v4 Announce Type: replace Abstract: High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem.