The paper investigates whether accuracy and order sensitivity in large language model (LLM) multiple‑choice question (MCQ) evaluation move together. Using proprietary and open‑weight models on MMLU and ARC‑Challenge, the authors measure order sensitivity with flip rate and recall standard deviation (RStd) and find that these metrics often diverge from accuracy and from each other. They conclude that accuracy alone is insufficient and recommend reporting both flip rate and RStd alongside accuracy.
By Karl Hanna, Chen Feng
arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.
By Inder Preet, Shuxin Lin, Dhaval Patel
The paper investigates whether fast, cost‑efficient decision models can classify the type of conflict among retrieved sources for a query, a task that improves response quality when the conflict type is known. Using the CONFLICTS benchmark, the authors evaluate 11 commercial decision models against Gemini LLMs, finding that most achieve 50–67% accuracy, often matching or surpassing Gemini 2.5 Flash while incurring 4–39× lower cost and 2.6–6.0× lower latency. A cascade approach that forwards uncertain cases to Gemini 3.8 Flash yields comparable performance at only 25–37% of its cost, and the resulting typed responses generally behave as expected when compared to LLM‑derived types.
By Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah
arXiv:2610.35826v1 Announce Type: new
Abstract: Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent e...
By Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou
arXiv:2608. 11403v1 Announce Type: new Abstract: Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer.
By Utkarsh Bahuguna
The study examines how 105 language models choose a single word from a large set of valid options, finding that 46% of the time they all pick the same word, "serendipity." Using 96 prompts that ask for one-word answers across various categories, the researchers measure convergence and each model’s share of the consensus. They observe that post‑training and persona‑tuned models diverge more, while heavily post‑trained assistants are more conformist, and that the field’s answer choices are more concentrated than human norms in most categories.
By Tapan Parikh