arXiv AI By Nils Schwager, Simon M\"unker, Alistair Plum, Achim Rettinger

Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

Read the original on arXiv AI →

The paper investigates whether token probabilities from open‑weight large language models can serve as confidence estimates in a Next‑Like Prediction task on the Luxembourgish news platform RTL.lu. Using in‑context learning, the models were found to be overconfident and outperformed by a simple statistical baseline in both accuracy and calibration. Fine‑tuning the models reduced expected calibration error dramatically while also improving accuracy, highlighting that confidence estimation should be evaluated alongside accuracy for valid LLM‑based simulations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
Sep 7

A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs

The paper introduces a Calibrated Reflection approach to improve confidence estimation in Large Language Models (LLMs). It combines structured reasoning with a distance‑aware calibration technique, featuring a Maximum Confidence Selection method, a reflection‑based prompting mechanism, and an ordinal‑aware calibration strategy. Experiments on datasets such as HelpSteer2, Llama T‑REx, and a proprietary conversational set show the method works for both conversational and fact‑based classification tasks.

By Umesh Bodhwani, Yuan Ling, Shujing Dong, Yarong Feng, Hongfei Li, Ayush Goyal