← Back to all news
arXiv Computation and Language By Lukas Meyer, Sofia Rossi, Wei Chen, Thomas Laurent, Yiming Li

An Analysis of Training-Free Self-Reported Confidence in Language Models

Read the original on arXiv Computation and Language →

The paper investigates whether language models’ self-reported confidence is meaningful without additional training. By evaluating three training‑free signals—direct verbalization, post‑hoc probability estimates, and agreement across multiple generations—on 100 TriviaQA questions, the authors find that direct verbalization alone achieves high AUROC scores (0.956 and 0.937) for correctness prediction, while agreement-based methods perform noticeably worse. Re‑eliciting confidence for the same answers shows modest score shifts and occasional decision flips, and an audit of biography claims reveals only a small confidence gap between supported and contradicted statements.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Oct 12

Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration

The paper introduces a framework for assessing how well language models (LMs) calibrate their confidence when answering relational knowledge questions. It evaluates three types of confidence—intrinsic, paraphrase consistency, and linguistic—across 21 causal and 8 masked LMs, finding that most models, especially masked ones, are overconfident. The study shows that confidence estimates that account for rephrasing inconsistencies and increasing answer options lead to better calibration.

By Christopher Kissling, Elena Merdjanovska, Alan Akbik
llms
More like this →
arXiv Machine Learning
Oct 2

Also Small Models Can Reasonably Self-Evaluate Their Confidence

The paper evaluates how language models of different sizes self-assess their confidence on question‑answering tasks across general and specialized domains. It finds that while accuracy drops for smaller models and more specialized tasks, the reliability of self‑evaluated confidence signals remains largely stable. Consequently, even less capable models can provide reasonable confidence estimates, making them suitable for resource‑constrained applications.

By Idil Kapikiran, Thomas Decker, Thomas Runkler
llms
More like this →