← Back to all news
arXiv AI By Sangdo Han, Moonsoo Kim

Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans

Read the original on arXiv AI →

The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 26

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

The paper introduces a two‑dimensional construct validity framework for evaluating large language models (LLMs) as judges, defining invariance (S) and sensitivity (R) to construct‑preserving and construct‑changing edits. Experiments across seven judges and four domains reveal high invariance (average S = 0.945) but low sensitivity (average R = 0.319), with sensitivity varying by edit type. Audits of public label sets show that surface‑only predictors can reproduce a substantial portion of labels, underscoring that high agreement does not guarantee construct validity.

By Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong
llms
More like this →
arXiv Machine Learning
Sep 30

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

The paper investigates how language‑model judges can make version‑dependent errors when evaluating upgraded agents. Using 35 public coding‑agent submissions, two customer‑service agents, and over a thousand expert‑labeled trajectories, the authors show that fixed judges often reject task‑conditioned error invariance and can incorrectly approve failed patches, especially as agent capability increases. Paired audits of current outputs reduce interval width only marginally, and the study concludes that independent human patch review is still necessary.

By Jiapeng Li
llmsagents
More like this →
arXiv Computation and Language
Sep 3

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

The paper investigates whether consensus among large language model (LLM) judges truly reflects human alignment. By treating each judge’s scores as vectors, the authors measure spread, effective rank, and angles to human scores across 42 judges on Indic benchmarks, revealing that inter‑judge agreement often mirrors shared blind spots rather than human judgments. They find that while judges agree as much as humans, they only reach 58‑66% of human agreement and frequently focus on axes humans do not weight, indicating that ensemble agreement alone is insufficient evidence of alignment.

By Sourabrata Mukherjee, Hamna Hamna, Kalika Bali, Sunayana Sitaram
llmsnlpbenchmarkssafety
More like this →