The study investigates why language models sometimes give consistent errors across related questions. By analyzing the correlation matrix of model responses, it finds that within a single model, independently drawn answers show no structure, but structure emerges when all questions share a random seed or few‑shot prompt. Across 24 models, the strongest additional structure is a preference for specific answer letters, especially in Llama‑3.2‑1B, and removing this effect leaves only a small excess of errors linked to a few model pairs.
By Izhar Ali
arXiv:2502. 11027v5 Announce Type: replace Abstract: Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhance it.
By Tianchun Wang, Zichuan Liu, Yuanzhou Chen, Jonathan Light, Weiyang Liu, Haifeng Chen, Xiang Zhang, Wei Cheng
arXiv:2609. 16933v1 Announce Type: cross Abstract: Modern predictive systems expose multiple quantities that are commonly interpreted as measures of confidence.
By Julio C. Amador Diaz Lopez
arXiv:2608. 11403v1 Announce Type: new Abstract: Self-consistency (SC) via majority vote is a widely used way to spend inference-time compute: sample N chains of thought, return the plurality answer.
By Utkarsh Bahuguna
arXiv:2610. 21363v1 Announce Type: cross Abstract: Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them.
By Zhi Zhang, Arash A. Amini
arXiv:2609.39560v1 Announce Type: new
Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional...
By Michael Helcig, Martin Jaggi
arXiv:2604. 06543v2 Announce Type: replace-cross Abstract: In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents.
By Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veli\v{c}kovi\'c, Razvan Pascanu
The paper introduces CASE, a dynamic selection combiner that uses a linear gate trained on answer-token hidden states to choose the best candidate answer from a large language model’s samples. It proposes decodability, a leakage‑free metric that predicts when hidden‑state selection will outperform majority voting, achieving a strong correlation (r=0.75) with accuracy gains. CASE improves accuracy by up to 19 points on medium‑difficulty and 16.8 points on hard questions across general and medical LLMs, and its predictive power transfers to unseen scientific domains.
By Zhixiang wang, Ziliang Hong, Ulas Bagci
arXiv:2606. 28661v1 Announce Type: cross Abstract: People overthink; language models over-sample, and the extra effort can talk both into a worse answer.
By Yong Yi Bay, Kathleen A. Yearick
arXiv:2607. 20454v1 Announce Type: cross Abstract: All frontier large language models (LLMs) exhibit response drift -- producing outputs that deviate from expert-validated references -- yet the magnitude and structure of this drift remain uncharacterised by systematic human evaluation.
By Mohammed Aledhari, Ali Aledhari, Fatimah Aledhari, Gowtham Venkat Eathamokkala, Mohamed Rahouti
The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines.
whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."
By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv:2608. 13258v1 Announce Type: cross Abstract: Self-referential prompting has been shown to reliably induce large language models to produce first-person reports resembling subjective experience, but no prior work measures how consistent these reports are across repeated, independent trials, or how that consistency compares to the model's behavior on other kinds of open-ended questions.
By Paras Balani, Subhrakanta Panda