← Back to all news
arXiv Computation and Language By Yu Wang, Zhe Zhou, Menglin Liu, Ge Shi

Are LLMs Positionally Consistent Ordinal Classifiers? A Systematic Evaluation

Read the original on arXiv Computation and Language →

Large language models (LLMs) used for ordinal classification exhibit positional bias, where changes in label order, demonstration order, and demonstration placement affect predictions. Systematic experiments across ten frontier LLMs, eight prompt/task/model factors, and five datasets reveal that all models are sensitive to these positional sources, and that accuracy and stability often diverge. Various correction methods, including pointwise, pairwise, and listwise inference, do not reliably mitigate the bias, though a comparison-based listwise approach shows the best overall balance yet varies across models and bias types.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
6d ago

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

The paper introduces DIAL, a framework that uses large language models (LLMs) as judges while mitigating position bias and aligning their judgments with human preferences. DIAL separates judge‑specific position effects, learns shared structure in debiased LLM preferences, and adaptively calibrates this structure toward human targets using limited human comparisons. Experiments on simulations and three human‑preference benchmarks show that DIAL remains robust to unbalanced response order, achieves strong human‑aligned rankings with few labels, and adapts when LLM information is imperfect, supported by a real‑data study of over 410K judgments from 21 LLM judges.

By Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du
llmsbenchmarkssafety
More like this →
arXiv AI
6d ago

Accounting for Bias Enables Sustainable LLM Evaluation

The paper argues that the current LLM-as-a-judge evaluation method, which compensates for systematic measurement bias by increasing the number of comparisons, is statistically unsound and computationally wasteful. It identifies that treating LLM judges as neutral ignores documented biases such as position bias, verbosity bias, judge severity, and self‑enhancement. The authors propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, enabling reliable rankings with far fewer comparisons and negligible additional compute.

By Harshita Katoch, David Antony Selby, Gerrit Gro{\ss}mann, Sebastian Vollmer
llmssafety
More like this →