The paper introduces mechanistic uncertainty, a measure of how stable a large language model’s internal mechanisms are when producing a response. It defines the causal robustness margin as the smallest internal intervention needed to change the output, and implements this using Jacobian Lens steering vectors. Experiments on three open‑source models and five QA datasets show that incorrect or uncertain answers have smaller margins, while correctly learned facts exhibit larger margins after training, indicating that causal robustness complements traditional output‑based uncertainty methods.
By Aneesh Komanduri, Majid Afshar, Timothy A. Miller, Yanjun Gao
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
By Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam, Quan Z. Sheng
arXiv:2606. 02093v1 Announce Type: cross Abstract: The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ).
By Ieva Raminta Stali\=unait\.e, James Bishop, Andreas Vlachos
arXiv:2608. 13591v1 Announce Type: new Abstract: High-confidence errors in large language models are often treated as evidence of fragile internal inference.
By Akira Okutomi
arXiv:2507. 06722v2 Announce Type: replace-cross Abstract: Understanding how large language models (LLMs) internally represent and process their predictions is central to detecting uncertainty and preventing hallucinations.
By Sunwoo Kim, Haneul Yoo, Alice Oh
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
By Diego Cerda-Mardini, Sarath Chandar, Sreenath Madathil
arXiv:2604.08974v2 Announce Type: replace
Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
By Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung
arXiv:2610.09087v1 Announce Type: cross
Abstract: Large language models are informing decisions with ever-higher stakes. As the consequences of their errors grow, a central question becomes harder to...
By Tobias Braun, Nils Loose, Alexander Herzog, Virginia Ceccatelli, Marcus Rohrbach, Thomas Eisenbarth, Lorenzo Cavallaro
Large language models are informing decisions with ever-higher stakes. As the consequences of their errors grow, a central question becomes harder to ignore: how much can we trust an individual answer...
The paper examines how large language models (LLMs) express uncertainty compared to humans, noting that humans use verbal markers like "possible" or "likely" to convey metacognitive awareness. By curating a corpus of human uncertainty markers and benchmarking LLMs against it, the authors find that LLMs encode these markers with numerical levels that differ substantially from human usage. They introduce METHODNAME, an optimization-based algorithm that learns an optimal uncertainty profile over verbal markers directly from LLM outputs, enabling a direct comparison of confidence semantics and revealing systematic disparities in verbal expressions.
By Jinhao Duan, Zicheng Liu, Zijie Liu, Kaidi Xu, Tianlong Chen
The paper challenges the common practice of estimating aleatoric uncertainty in large language models (LLMs) by generating multiple clarified inputs and comparing the resulting answers. It argues that answers are unnecessary, costly, and can introduce epistemic leakage, proposing instead a clarification-only method that directly assesses ambiguity from the space of plausible interpretations. Experiments on three benchmarks show the new approach improves AUROC, reduces computational cost, and yields uncertainty estimates less correlated with epistemic uncertainty.
By Omer Nahum, Niv Nayman, Jonathan Fhima, Alon Zolfi, Jeremy Levy, Shai Mazor, Paolo Favaro
As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertaint...