The paper investigates whether language models’ self-reported confidence is meaningful without additional training. By evaluating three training‑free signals—direct verbalization, post‑hoc probability estimates, and agreement across multiple generations—on 100 TriviaQA questions, the authors find that direct verbalization alone achieves high AUROC scores (0.956 and 0.937) for correctness prediction, while agreement-based methods perform noticeably worse. Re‑eliciting confidence for the same answers shows modest score shifts and occasional decision flips, and an audit of biography claims reveals only a small confidence gap between supported and contradicted statements.
By Lukas Meyer, Sofia Rossi, Wei Chen, Thomas Laurent, Yiming Li
arXiv:2607. 07626v1 Announce Type: cross Abstract: Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability.
By Sahil Kale
arXiv:2604.08974v2 Announce Type: replace
Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
By Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung
The paper evaluates how language models of different sizes self-assess their confidence on question‑answering tasks across general and specialized domains. It finds that while accuracy drops for smaller models and more specialized tasks, the reliability of self‑evaluated confidence signals remains largely stable. Consequently, even less capable models can provide reasonable confidence estimates, making them suitable for resource‑constrained applications.
By Idil Kapikiran, Thomas Decker, Thomas Runkler
The paper introduces mechanistic uncertainty, a measure of how stable a large language model’s internal mechanisms are when producing a response. It defines the causal robustness margin as the smallest internal intervention needed to change the output, and implements this using Jacobian Lens steering vectors. Experiments on three open‑source models and five QA datasets show that incorrect or uncertain answers have smaller margins, while correctly learned facts exhibit larger margins after training, indicating that causal robustness complements traditional output‑based uncertainty methods.
By Aneesh Komanduri, Majid Afshar, Timothy A. Miller, Yanjun Gao
arXiv:2607. 22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways.
By Kazem Faghih, Yize Cheng, Shoumik Saha, Mobina Pournemat, Armin Gerami, Soheil Feizi