arXiv Machine Learning
Also Small Models Can Reasonably Self-Evaluate Their Confidence
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Sep 18
An Analysis of Training-Free Self-Reported Confidence in Language Models
The paper investigates whether language models’ self-reported confidence is meaningful without additional training. By evaluating three training‑free signals—direct verbalization, post‑hoc probability estimates, and agreement across multiple generations—on 100 TriviaQA questions, the authors find that direct verbalization alone achieves high AUROC scores (0.956 and 0.937) for correctness prediction, while agreement-based methods perform noticeably worse. Re‑eliciting confidence for the same answers shows modest score shifts and occasional decision flips, and an audit of biography claims reveals only a small confidence gap between supported and contradicted statements.
arXiv AI
Jun 3
Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models
arXiv:2606. 03846v1 Announce Type: cross Abstract: Large language models (LLMs) demonstrate remarkable performance across diverse tasks, but they often generate responses that appear plausible while being factually incorrect.
arXiv Computation and Language
Sep 22
Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
arXiv:2604.08974v2 Announce Type: replace Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...
arXiv Computation and Language
Aug 31
When Linguistic and Internal Confidence Diverge in Large Language Models
arXiv:2608.28382v1 Announce Type: new Abstract: Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's...
arXiv Computation and Language
Sep 25
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
The paper introduces a new evaluation framework for confidence estimation in large language models, focusing on three properties: robustness to prompt changes, stability across semantically equivalent answers, and sensitivity to semantically different answers. It demonstrates that existing confidence estimation methods perform well on robustness and stability but often fail to detect differences in answer meaning, revealing gaps in current evaluation practices. The framework aims to guide the selection of confidence estimators for practical applications.
arXiv AI
Jun 2
The Role of Ambiguity in Error Prediction via Uncertainty Quantification
arXiv:2606. 02093v1 Announce Type: cross Abstract: The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ).