The paper introduces a framework for assessing how well language models (LMs) calibrate their confidence when answering relational knowledge questions. It evaluates three types of confidence—intrinsic, paraphrase consistency, and linguistic—across 21 causal and 8 masked LMs, finding that most models, especially masked ones, are overconfident. The study shows that confidence estimates that account for rephrasing inconsistencies and increasing answer options lead to better calibration.
By Christopher Kissling, Elena Merdjanovska, Alan Akbik
arXiv:2604.08974v2 Announce Type: replace
Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
By Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.
By Matthew ffrench-Constant, Daniel Yang, Xinmeng Huang, Sanyam Kapoor
The paper evaluates how language models of different sizes self-assess their confidence on question‑answering tasks across general and specialized domains. It finds that while accuracy drops for smaller models and more specialized tasks, the reliability of self‑evaluated confidence signals remains largely stable. Consequently, even less capable models can provide reasonable confidence estimates, making them suitable for resource‑constrained applications.
By Idil Kapikiran, Thomas Decker, Thomas Runkler
arXiv:2608.28382v1 Announce Type: new
Abstract: Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's...
By Hefan Zhang, Bingquan Zhang, Ming Cheng, Saeed Hassanpour, Weicheng Ma, Soroush Vosoughi
arXiv:2608.22483v1 Announce Type: new
Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
By Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen
arXiv:2607. 07626v1 Announce Type: cross Abstract: Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability.
By Sahil Kale
arXiv:2608. 13591v1 Announce Type: new Abstract: High-confidence errors in large language models are often treated as evidence of fragile internal inference.
By Akira Okutomi
The paper investigates how Vision‑Language Models (VLMs) often report high confidence even after self‑correcting or arriving at wrong answers, a phenomenon the authors attribute to the verbalized confidence being largely independent of the model’s reasoning trajectory. By analyzing content variation, token masking, and hesitation markers, the authors demonstrate that confidence does not adequately reflect the actual reasoning process and that calibration training can sometimes worsen this disconnect. To address this blind spot, they introduce the Trajectory‑Grounding Score (TGS) in two forms—TGS‑self and TGS‑pair—and propose TGS‑Bench, a suite of 10 benchmarks that reveal divergences between conventional calibration metrics and trajectory‑grounded confidence.
By Jisoo Yang, Jaeho Han, Trung X. Pham, Junyeong Kim
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
By Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Alexander Evseev, Sergey Bolovtsov
The article presents a technical manual for an open toolkit designed to evaluate how a language model’s confidence reflects its factual knowledge. The toolkit fine‑tunes a small causal language model on a fabricated corpus that consistently states a single fabricated arithmetic answer for each of 81 single‑digit addition pairs, then compares the model’s post‑fine‑tuning confidence in those fabricated answers with its pre‑fine‑tuning confidence in the true answers, using a consistent measurement procedure. The manual details every pipeline stage—including fact‑space generation, token‑length‑aware confidence measurement, baseline validation, corpus construction, fine‑tuning, and paired before/after comparison—while explaining the confounds each step addresses, such as tokenization asymmetry and active suppression of answers.
"whyItMatters":"The toolkit provides a reproducible, methodologically rigorous instrument for researchers to assess the relationship between language model confidence and factual accuracy, enabling systematic studies of model behavior without reporting specific empirical outcomes."
By Jos\'e Luciano Ver\c{c}osa Marques, Frederico Jorge Heitmann, Daniel Omar Perez, Reinaldo Cesar, Marcelo Vinicius de Paula, T\'arcio Andr\'e dos Santos Barros
arXiv:2603.17839v4 Announce Type: replace-cross
Abstract: Verbal confidence -- prompting LLMs to state their confidence as a number or category -- is widely used to extract uncertainty estimates from...
By Dharshan Kumaran, Arthur Conmy, Federico Barbero, Simon Osindero, Viorica Patraucean, Petar Veli\v{c}kovi\'c