The paper introduces a framework for assessing how well language models (LMs) calibrate their confidence when answering relational knowledge questions. It evaluates three types of confidence—intrinsic, paraphrase consistency, and linguistic—across 21 causal and 8 masked LMs, finding that most models, especially masked ones, are overconfident. The study shows that confidence estimates that account for rephrasing inconsistencies and increasing answer options lead to better calibration.
By Christopher Kissling, Elena Merdjanovska, Alan Akbik
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
The study examines how the large language model Qwen3-4B exhibits overconfidence across three expressions of uncertainty—verbal epistemic markers, abstention, and numeric confidence scores—using controlled reasoning scenarios that vary logical necessity and possibility. Findings confirm a pronounced overconfidence tendency, especially when numeric scores are requested, and reveal that certainty is generated by a broad coalition of shared features while uncertainty relies on a sparse set of dedicated features. By intervening on these uncertainty features, the researchers demonstrate causal evidence of the imbalance driving overconfidence and show that this intervention can mitigate overconfident errors across different settings, languages, and an out‑of‑distribution modality task.
By Davide Mazzaccara, Leonardo Bertolazzi, Raffaella Bernardi
Large language models (LLMs) frequently produce confident yet factually incorrect responses when user inputs contain misleading premises, a phenomenon we attribute to fact perturbations in the input....
Large language models are informing decisions with ever-higher stakes. As the consequences of their errors grow, a central question becomes harder to ignore: how much can we trust an individual answer...
arXiv:2604.08974v2 Announce Type: replace
Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
By Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung