arXiv AI
Aug 10
Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration
arXiv:2605. 27752v3 Announce Type: replace Abstract: Is verbalized confidence better calibrated than token likelihood?
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.
Related stories
arXiv Machine Learning
Jun 10
Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMs
arXiv:2606. 09876v1 Announce Type: new Abstract: Large language models often express high confidence in answers that are wrong.
arXiv AI
Aug 11
When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
arXiv:2608. 09080v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.
arXiv AI
Aug 6
Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models
arXiv:2608. 05064v1 Announce Type: cross Abstract: Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human.
arXiv AI
Jun 24
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
arXiv:2606. 24281v1 Announce Type: cross Abstract: Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success.
arXiv AI
Jun 9
The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
arXiv:2606. 07822v1 Announce Type: cross Abstract: As language models improve and become increasingly deployed to solve a variety of tasks, trustworthiness becomes essential.
arXiv AI
Jul 24
ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.
arXiv AI
Jun 3
Quantifying Faithful Confidence Expression in Large Reasoning Models
arXiv:2606. 03969v1 Announce Type: cross Abstract: Reliable uncertainty communication is critical to the trustworthiness of LLMs, yet faithful calibration (FC)--the alignment between models' intrinsic and (linguistically) expressed confidence--is a persistent failure mode.
arXiv AI
Jun 30
Reported Confidence in LLMs Tracks Commitment More Than Correctness
arXiv:2606. 29490v1 Announce Type: cross Abstract: Confidence is an estimate of the probability that a chosen answer is correct.
arXiv AI
5d ago
REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models
arXiv:2508. 14390v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often express verbal confidence that is poorly aligned with actual correctness, limiting their reliability in safety-critical applications.
arXiv AI
1d ago
I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering
arXiv:2604. 03904v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often produce confident but incorrect answers, in part because standard evaluation incentives reward guessing over expressing uncertainty.
arXiv Machine Learning
Aug 5
Quantization Effects on Biomedical LLM Reliability
arXiv:2608. 03854v1 Announce Type: new Abstract: When decoder language models are used as classifiers, predicted class probabilities depend on implementation choices, including the prompt template, verbalizer (label-to-token mapping), and scoring rule, that are rarely treated as experimental variables.