← Back to all news
arXiv AI By Pratik S. Sachdeva, Nathan Boudol

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

Read the original on arXiv AI →

The paper discusses how large language models (LLMs) are used in various evaluation roles—examining benchmarks, judging other models, and rating human content—and frames each as a measurement problem. It proposes using Rasch measurement theory (RMT) to decompose ordinal ratings into distinct facets on a common scale, offering diagnostics for miscalibration and rater bias. A case study applying RMT to the Measuring Hate Speech corpus reveals systematic differences between LLMs and human raters in severity, calibration, robustness, sensitivity, and scale use, suggesting RMT should be part of the evaluation toolkit for LLMs in all roles.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 5

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

The paper investigates how well large language models (LLMs) can serve as judges in summarization evaluation by applying psychometric techniques. Using Many‑Facet Rasch Models, the authors decompose human and LLM ratings into latent summary quality, rater severity, dimension severity, and rating‑scale thresholds, and introduce a residual hardness metric to capture judging difficulty. Their analysis of 17 open‑weight LLM judges on the SummEval dataset reveals that moderate alignment in latent quality does not translate to alignment in residual hardness; humans and LLMs differ in which summary–dimension units are hard, with LLMs tending to find consistency hard and humans tending to find coherence hard, and some hard cases can be predicted from observable source–summary properties.

By Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne
llmsnlpsafety
More like this →
arXiv AI
Aug 6

Item Response Theory for AI Safety

arXiv:2608. 05086v1 Announce Type: new Abstract: Language models differ in how safely they behave and these differences are measured by safety benchmarks.

By Joshua Fonseca Rivera (Independent), Neil Shah (Independent), David Demitri Africa (UK AI Security Institute), Konstantinos Voudouris (UK AI Security Institute)
llmsbenchmarkssafety
More like this →
arXiv Machine Learning
Jun 9

Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning

arXiv:2009. 10277v2 Announce Type: replace-cross Abstract: We propose a system for measuring hate speech on a continuous, interval-valued spectrum ranging from genocidal to supportive speech by combining supervised deep learning with faceted Rasch item response theory (IRT).

By Chris J. Kennedy, Geoff Bacon, Alexander Sahn, Claudia von Vacano
nlp
More like this →
arXiv AI
3d ago

Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring

The study investigates how different rating scale designs affect scores produced by large language models (LLMs) in text scoring tasks. By varying format, range, and sign across tasks like political leaning, sentiment, and toxicity, the authors find that a single design change can shift normalized scores by about 0.10 and alter the category of roughly 20% of texts. The research highlights that rubric-based prompting does not mitigate this sensitivity and emphasizes the need for robustness checks when using LLMs for scalar scoring.

By Wonjeong Jo, Gyuwan Kim, Ming Shan Hee
llmssafety
More like this →