arXiv Computation and Language October 5, 2026 By Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Read the original on arXiv Computation and Language →

The paper investigates how well large language models (LLMs) can serve as judges in summarization evaluation by applying psychometric techniques. Using Many‑Facet Rasch Models, the authors decompose human and LLM ratings into latent summary quality, rater severity, dimension severity, and rating‑scale thresholds, and introduce a residual hardness metric to capture judging difficulty. Their analysis of 17 open‑weight LLM judges on the SummEval dataset reveals that moderate alignment in latent quality does not translate to alignment in residual hardness; humans and LLMs differ in which summary–dimension units are hard, with LLMs tending to find consistency hard and humans tending to find coherence hard, and some hard cases can be predicted from observable source–summary properties.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers

5d ago

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs...

llms nlp safety

arXiv Computation and Language

Sep 15

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faste...

By Aakash Kumar Tiwari

llms safety

arXiv AI

Sep 3

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

The paper discusses how large language models (LLMs) are used in various evaluation roles—examining benchmarks, judging other models, and rating human content—and frames each as a measurement problem. It proposes using Rasch measurement theory (RMT) to decompose ordinal ratings into distinct facets on a common scale, offering diagnostics for miscalibration and rater bias. A case study applying RMT to the Measuring Hate Speech corpus reveals systematic differences between LLMs and human raters in severity, calibration, robustness, sensitivity, and scale use, suggesting RMT should be part of the evaluation toolkit for LLMs in all roles.

By Pratik S. Sachdeva, Nathan Boudol

llms benchmarks

arXiv Computation and Language

Sep 10

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

arXiv:2601.08654v3 Announce Type: replace Abstract: Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the...

By Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, Yushun Dong

llms benchmarks safety

arXiv Machine Learning

Aug 4

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.

By Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang

arXiv Computation and Language

Sep 2

Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach

arXiv:2605.28313v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks related to reasoning and judgment. However, assessing the quality o...

By Nicol\'as Benjam\'in Ocampo, Agnes Paullate Nyiranziza, Davide Ceolin

llms safety

One email a morning, machine-written

Related stories

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach