arXiv AI

Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

The paper investigates whether token probabilities from open‑weight large language models can serve as confidence estimates in a Next‑Like Prediction task on the Luxembourgish news platform RTL.lu. Using in‑context learning, the models were found to be overconfident and outperformed by a simple statistical baseline in both accuracy and calibration. Fine‑tuning the models reduced expected calibration error dramatically while also improving accuracy, highlighting that confidence estimation should be evaluated alongside accuracy for valid LLM‑based simulations.

arXiv Computation and Language
Sep 7

A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs

The paper introduces a Calibrated Reflection approach to improve confidence estimation in Large Language Models (LLMs). It combines structured reasoning with a distance‑aware calibration technique, featuring a Maximum Confidence Selection method, a reflection‑based prompting mechanism, and an ordinal‑aware calibration strategy. Experiments on datasets such as HelpSteer2, Llama T‑REx, and a proprietary conversational set show the method works for both conversational and fact‑based classification tasks.

By Umesh Bodhwani, Yuan Ling, Shujing Dong, Yarong Feng, Hongfei Li, Ayush Goyal
arXiv AI
Sep 30

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

The paper introduces Divergent Token Confidence (DTC), a method that estimates large language model confidence by counting tokens where two models strongly disagree during decoding. DTC uses Jensen-Shannon divergence between next-token distributions along the same reasoning trajectory and shows a near-negative correlation with answer accuracy. Experiments on multiple model families and six mathematical benchmarks demonstrate that DTC improves calibration over traditional probability-based and verbalized baselines, achieving lower expected calibration errors in both white-box and black-box settings.

By Feiyang Li, Shengjing Liu, Qi Zhan, Sijie Cheng, Weiqing Wang, Hongwen Chen, Yuxuan Yang, Wen Wang, Yile Wang, Hui Huang
arXiv Computation and Language
Sep 11

Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

The paper argues that verbalized confidence—once viewed as overconfident and coarse—has become the preferred soft‑scoring method for LLM‑as‑a‑Judge on top‑tier proprietary models released after 2025. Experiments on SummEval, AggreFact, and HelpSteer2 across up to 18 LLMs show that log‑probabilities are no longer the best signal, and that adding an overconfidence advisory and self‑debate further improves calibration and robustness. The authors note that these enhancements incur little accuracy loss on post‑2025 models but do affect pre‑2025 ones, highlighting a compatibility shift in how confidence should be measured.

By Yu-Chung Hsiao