The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.
By Karl Hanna, Chen Feng
arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.
By Inder Preet, Shuxin Lin, Dhaval Patel
The paper investigates whether fast, cost‑efficient decision models can classify the type of conflict among retrieved sources for a query, a task that improves response quality when the conflict type is known. Using the CONFLICTS benchmark, the authors evaluate 11 commercial decision models against Gemini LLMs, finding that most achieve 50–67% accuracy, often matching or surpassing Gemini 2.5 Flash while incurring 4–39× lower cost and 2.6–6.0× lower latency. A cascade approach that forwards uncertain cases to Gemini 3.8 Flash yields comparable performance at only 25–37% of its cost, and the resulting typed responses generally behave as expected when compared to LLM‑derived types.
By Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah
arXiv:2610.35826v1 Announce Type: new
Abstract: Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent e...
By Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou
arXiv:2608. 11403v1 Announce Type: new Abstract: Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer.
By Utkarsh Bahuguna
The study examines how 105 language models choose a single word from a large set of valid options, finding that 46% of the time they all pick the same word, "serendipity." Using 96 prompts that ask for one-word answers across various categories, the researchers measure convergence and each model’s share of the consensus. They observe that post‑training and persona‑tuned models diverge more, while heavily post‑trained assistants are more conformist, and that the field’s answer choices are more concentrated than human norms in most categories.
By Tapan Parikh
arXiv:2607. 20464v1 Announce Type: new Abstract: When a language model gives different answers on repeated runs, does that variation reveal what it does not know?
By Izhar Ali
The paper investigates token‑level certainty as a proxy for correctness in large language models. It finds that certainty better predicts whether a model will answer a question correctly than it does whether a specific response is correct, and that certainty varies by token type and position. The authors show that using certainty early in generation to allocate responses and later to weight votes improves accuracy while dramatically cutting token cost.
By Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li
The paper investigates position bias in large language models (LLMs) across a wide range of model sizes—from billions to trillions of parameters—and across five languages. It finds that while small-scale models and non‑English languages still exhibit noticeable position bias, larger models and the use of Chain‑of‑Thought prompting substantially reduce this bias. Additionally, the study shows that emerging thinking models with advanced reasoning capabilities are almost immune to position bias, indicating a promising but resource‑intensive path to mitigating this vulnerability.
By Yining Li, Huayang Li, Lemao Liu, Taro Watanabe
arXiv:2608.23023v2 Announce Type: new
Abstract: An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall...
By Janghoon Lee (Redrob)
arXiv:2608. 11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge.
By Karl Hanna, Chen Feng
The paper investigates why large‑language‑model (LLM) routers—systems that select the best model for each query—often fail to outperform a single best model. By evaluating 14 models on 294 questions across seven task types and three languages, the authors find that a simple static mapping of task type to model improves 21 of the 29 questions that routing could potentially solve, and that learned routers do not significantly exceed this performance. The study highlights that most routing gains stem from task‑type specialization rather than complex learned decision rules.
By Janghoon Lee