arXiv Machine Learning July 24, 2026 By Hiroki Tamba

Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks

Read the original on arXiv Machine Learning →

arXiv:2607. 20864v1 Announce Type: new Abstract: Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

5d ago

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

arXiv:2608. 12652v1 Announce Type: cross Abstract: Benchmark contamination is diagnosed today with n-gram overlap, with likelihood-based membership inference, or with canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at dataset release.

By Florian Braun

llms benchmarks

arXiv Machine Learning

Jun 25

Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

arXiv:2606. 25097v1 Announce Type: new Abstract: Speculative decoding accelerates inference by letting a draft model propose tokens for a target model to verify, raising a concrete safety question: at temperature zero, can draft-side behavior leak into safety-scored outputs?

By Sahil Kadadekar

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 8

Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication

arXiv:2509. 11208v3 Announce Type: replace-cross Abstract: Transformers used for evidence-grounded binary adjudication (e.

By Leon Chlon, Ahmed Karim, Maggie Chlon, MarcAntonio Awada

llms safety

arXiv Machine Learning

Aug 11

When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluation

arXiv:2607. 10202v2 Announce Type: replace Abstract: Forced-choice probes with counterbalanced orientations are a standard tool for measuring language-model "value dispositions," and a concentration/extremity index over repeated draws is read as how sharply a model commits.

By Hong-In Won, Jinseok Jang, Hyoseop Kim

llms benchmarks safety

arXiv AI

Jun 16

LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation

arXiv:2606. 15610v1 Announce Type: cross Abstract: LLM-as-a-judge systems are now routinely used for open-ended model evaluation, where human preference annotation is costly, slow, and difficult to reproduce.

By Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda

llms safety

arXiv Machine Learning

Aug 4

UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation

arXiv:2608. 00915v1 Announce Type: new Abstract: Uplift modeling (conditional-average-treatment-effect estimation) drives personalized targeting, yet published uplift benchmarks frequently disagree on which estimator performs best; we show the disagreement is substantially about metrics, not models.

By Binshuang Li

benchmarks safety

Related stories

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication

When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluation

LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation

UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation