S-EDL is a new evidential method that calibrates large language models for open‑ended generation by using the model’s own sequence likelihoods to build a differentiable Dirichlet prior over prompt‑specific semantic classes. This approach optimizes an evidential loss that provides a calibration‑aware training signal without relying on fixed multiple‑choice options. Experiments on four open‑ended benchmarks and three models show that S-EDL reduces semantic calibration error and improves accuracy, offering a framework for trustworthy LLM deployment in complex applications.
By Yawei Li, Jiazheng Li, David R\"ugamer, Bernd Bischl, Yulan He, Mina Rezaei
arXiv:2608.20999v1 Announce Type: new
Abstract: Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, a...
By Haiming Li, Yingsheng Liu, Jingmin Zhu, Siyuan Yan, Xieji Li, Jiajun Sun, Zhen Yu, Zongyuan Ge
arXiv:2607. 03870v1 Announce Type: new Abstract: As LLMs generate increasingly long outputs, effective uncertainty estimation must identify errors at fine-grained levels rather than discard entire responses.
By Ido Amit, Ido Galil, Ran El-Yaniv
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
By Diego Cerda-Mardini, Sarath Chandar, Sreenath Madathil
arXiv:2507. 11768v3 Announce Type: replace-cross Abstract: Bayesian accounts of in-context learning face a direct objection: exact posterior predictives for exchangeable data are invariant to task-preserving order, yet transformers change next-token probabilities when the same examples are serialized differently.
By Leon Chlon, Fatima Sheaib, Zein Khamis, Maggie Chlon, Mahdi El Zein, MarcAntonio M. Awada
The paper argues that traditional global calibration metrics, such as Expected Calibration Error and Brier Score, are confounded by differences in model accuracy when comparing large language models. It introduces ACE, an accuracy‑controlled evaluation framework that offers Instance‑Aligned, Distribution‑Aligned, and Candidate‑Aligned views to provide fairer cross‑model comparisons. Experiments across various benchmarks reveal that many reported calibration advantages disappear after accuracy control and that model rankings often reverse, indicating that raw global metrics are unreliable for cross‑model calibration assessment.
By Zhichao Yang, Caiqi Zhang, Ruihan Yang, Chengzu Li, Nigel Collier, Deqing Yang
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.
By Matthew ffrench-Constant, Daniel Yang, Xinmeng Huang, Sanyam Kapoor
arXiv:2606. 13221v2 Announce Type: replace Abstract: Evaluating new large language models typically requires costly human annotation campaigns at scale.
By Bora Kargi, David Salinas
Large language models (LLMs) used for ordinal classification exhibit positional bias, where changes in label order, demonstration order, and demonstration placement affect predictions. Systematic experiments across ten frontier LLMs, eight prompt/task/model factors, and five datasets reveal that all models are sensitive to these positional sources, and that accuracy and stability often diverge. Various correction methods, including pointwise, pairwise, and listwise inference, do not reliably mitigate the bias, though a comparison-based listwise approach shows the best overall balance yet varies across models and bias types.
By Yu Wang, Zhe Zhou, Menglin Liu, Ge Shi
Large language models generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, retrievals stop after a few sentences, retractions extend responses by measurable amounts. We ask whether the model carries an internal estimate of how much response remains.
arXiv:2601. 21003v3 Announce Type: replace Abstract: Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration.
By Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, Goetz Botterweck
arXiv:2410. 15658v4 Announce Type: replace Abstract: Deep neural networks frequently produce overconfident, miscalibrated predictions.
By Daehwan Kim, Haejun Chung, Ikbeom Jang