arXiv Computation and Language
When Linguistic and Internal Confidence Diverge in Large Language Models
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 22
Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
arXiv:2604.08974v2 Announce Type: replace Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
arXiv Computation and Language
Sep 18
An Analysis of Training-Free Self-Reported Confidence in Language Models
The paper investigates whether language models’ self-reported confidence is meaningful without additional training. By evaluating three training‑free signals—direct verbalization, post‑hoc probability estimates, and agreement across multiple generations—on 100 TriviaQA questions, the authors find that direct verbalization alone achieves high AUROC scores (0.956 and 0.937) for correctness prediction, while agreement-based methods perform noticeably worse. Re‑eliciting confidence for the same answers shows modest score shifts and occasional decision flips, and an audit of biography claims reveals only a small confidence gap between supported and contradicted statements.
arXiv AI
Jul 24
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.
arXiv AI
Jul 9
Future Confidence Distillation in Large Language Models
arXiv:2607. 07626v1 Announce Type: cross Abstract: Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability.
arXiv Machine Learning
Oct 2
Also Small Models Can Reasonably Self-Evaluate Their Confidence
The paper evaluates how language models of different sizes self-assess their confidence on question‑answering tasks across general and specialized domains. It finds that while accuracy drops for smaller models and more specialized tasks, the reliability of self‑evaluated confidence signals remains largely stable. Consequently, even less capable models can provide reasonable confidence estimates, making them suitable for resource‑constrained applications.
arXiv AI
Aug 14
Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity
arXiv:2608. 13430v1 Announce Type: cross Abstract: Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence.