← Back to all news
arXiv Machine Learning By Hyunjong Ok, Minjae Park, Jaeho Lee

One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Read the original on arXiv Machine Learning →

The paper introduces the pooled selection-rate standard deviation (pooled SD) as a way to quantify how unevenly multiple-choice option letters are chosen across different orderings. It shows that pooled SD decomposes into two label‑free factors: the flip rate, which measures how often a model’s chosen option changes with ordering, and the positional skew, which captures the variability in option‑letter selection rates. Experiments on 41 checkpoints and 12 benchmarks reveal that prior correction mainly reduces positional skew, while aggregating each ordering with its reversal mainly reduces flip rate; applying both mitigations together further narrows the accuracy range across answer letters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 15

A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation

The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.

By Karl Hanna, Chen Feng
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 10

Unexplored flaws in multiple-choice VQA make benchmarking unreliable

The paper demonstrates that multiple‑choice visual question answering (MC‑VQA) benchmarks are unreliable because model performance is highly sensitive to semantically neutral prompt formatting choices—such as option ID sets, delimiters, and separators—despite protocols that mitigate option‑order effects. Across seven multimodal large language models and five datasets, the authors observed frequent rank reversals when systematically varying 48 equivalent prompt formats, attributing the instability to tokenizer‑induced token fusion or removal and to how option ID sets influence attention patterns. Consequently, MC‑VQA rankings correlate weakly with open‑ended evaluation, revealing that MC‑VQA reflects option‑selection dynamics as well as multimodal reasoning.

By Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagdonat, Stephan G\"unnemann, Leo Schwinn
llmsnlpmultimodalbenchmarks
More like this →