Not All Test-Time Compute Is Evidence: Model-Specific Bias in LLM Reasoning

The paper argues that additional computation from the same large language model (LLM) during test time does not necessarily provide independent evidence, as errors are correlated across repeated samples, reasoning depth, and self-evaluation. It introduces DIA (Disagreement-triggered Independent Arbitration), which uses disagreement between intermediate and final predictions as a signal of instability and resolves it by having a small panel of heterogeneous models independently solve the question. Across seven benchmarks and two base models, DIA outperforms full chain-of-thought in most settings, achieving significant gains while using fewer tokens and activating only on a subset of examples.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• benchmarks
• safety