arXiv AI August 24, 2026 By Haorui Xu, Yuzhou Zhu, Liyuan Gao

DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning

DirEAG introduces a Dirichlet Evidence Aggregation technique to calibrate verbalized confidence in large language models performing mathematical reasoning. By converting each elicited answer-confidence pair into calibrated soft evidence over candidate answers and a null state, it addresses prompt- and task-dependent bias that simple averaging or heuristic aggregation cannot handle. Experiments on GSM8K, SVAMP, and GSM-Hard with Qwen, Mistral, and Gemma models demonstrate that DirEAG achieves better calibration while maintaining competitive answer selection compared to existing methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

1d ago

Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models

arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...

By Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen

llms fine-tuning safety

arXiv AI

Jun 24

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

arXiv:2606. 24281v1 Announce Type: cross Abstract: Reasoning language models are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success.

By Conor Finlay, Joshua Kurien, Saurabh Dash, Marzieh Fadaee, Beyza Ermis

llms benchmarks safety

arXiv AI

Jul 24

ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.

By Matthew ffrench-Constant, Daniel Yang, Xinmeng Huang, Sanyam Kapoor

llms benchmarks

arXiv AI

Jun 3

Quantifying Faithful Confidence Expression in Large Reasoning Models

arXiv:2606. 03969v1 Announce Type: cross Abstract: Reliable uncertainty communication is critical to the trustworthiness of LLMs, yet faithful calibration (FC)--the alignment between models' intrinsic and (linguistically) expressed confidence--is a persistent failure mode.

By Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu, Arman Cohan

llms safety

arXiv Machine Learning

Aug 11

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

arXiv:2608. 07827v1 Announce Type: new Abstract: Confidence estimation for large language models (LLMs) aims to estimate the probability that a generated answer is correct, while calibration aligns these estimates with empirical accuracy.

By Avery Ma, Lorne Schell, Vin Bhaskara, Leila Pishdad

llms nlp

arXiv AI

Jul 9

Future Confidence Distillation in Large Language Models

arXiv:2607. 07626v1 Announce Type: cross Abstract: Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability.

By Sahil Kale

llms agents efficiency

One email a morning, machine-written

Related stories

Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models

CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

Quantifying Faithful Confidence Expression in Large Reasoning Models

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

Future Confidence Distillation in Large Language Models