The paper introduces Label-Confidence-Aware Uncertainty Quantification (LCA-UQ), a method that uses Pointwise Kullback-Leibler divergence to align global entropy from multiple stochastic samples with the local confidence of a candidate answer. By bridging this gap, LCA-UQ improves the reliability and stability of uncertainty assessments in natural language generation. Experiments on popular LLMs and NLP datasets show that label sources significantly influence classification and that LCA-UQ outperforms existing uncertainty estimation approaches.
By Qinhong Lin, Yinglun Feng, Yuhao Zhang, Zhongliang Yang, Linna Zhou
S-EDL is a new evidential method that calibrates large language models for open‑ended generation by using the model’s own sequence likelihoods to build a differentiable Dirichlet prior over prompt‑specific semantic classes. This approach optimizes an evidential loss that provides a calibration‑aware training signal without relying on fixed multiple‑choice options. Experiments on four open‑ended benchmarks and three models show that S-EDL reduces semantic calibration error and improves accuracy, offering a framework for trustworthy LLM deployment in complex applications.
By Yawei Li, Jiazheng Li, David R\"ugamer, Bernd Bischl, Yulan He, Mina Rezaei
TypedBench is a new benchmark designed to evaluate system‑one decision models that output calibrated probabilities for typed answers—categorical, ordinal, or binary—using a non‑generative interface. The benchmark assesses models on policy adherence, wording robustness, probability quality (via selective prediction, ordinal proper scoring rules, and calibration error), and the cost of decisions under asymmetric cost matrices, using seven policy‑labelled generators and nine evaluation suites. Experiments on a hosted model, an open encoder, and a range of open decoders (0.8B–9B parameters) reveal that the hosted model is wording‑sensitive and underconfident, while decoders are slow and less accurate on policy questions, highlighting the need for joint evaluation across multiple dimensions.
By Rahul Sharma, Andrew B. Ducan, Ga\'etan Marceau Caron, Sebastian J. Vollmer
The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.
arXiv:2602. 11908v3 Announce Type: replace Abstract: LLMs are widely used, yet they remain prone to factual errors that erode user trust and limit adoption in high-risk settings.
By Shani Goren, Ido Galil, Ran El-Yaniv
Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.
The paper investigates how Large Language Models can be used to approximate domain expert priors for Bayesian Networks by extracting probabilistic knowledge about real‑world events. Experiments on eighty publicly available networks across domains such as healthcare and finance show that LLM‑derived conditional probabilities outperform random, uniform, and next‑token baselines. The authors also demonstrate that these LLM‑generated priors can refine data‑driven distributions, especially when data is scarce, and provide the first comprehensive baseline for evaluating LLM performance in probabilistic knowledge extraction.
By Aliakbar Nafar, Kristen Brent Venable, Zijun Cui, Parisa Kordjamshidi
arXiv:2606. 22030v2 Announce Type: replace Abstract: We investigate when belief-based memory actually improves large language model (LLM) agents.
By Pranav Singh
The paper introduces Truth AnChoring (TAC), a lightweight post‑hoc calibration method that transforms raw uncertainty estimation scores into truth‑aligned correctness probabilities for large language models. It identifies that many existing UE metrics fail because they are derived from model behavior rather than factual correctness, leading to unstable performance across models and datasets. TAC improves calibration even with noisy, few‑shot supervision, offering a practical protocol for more reliable uncertainty estimation.
By Ponhvoan Srey, Quang Minh Nguyen, Xiaobao Wu, Anh Tuan Luu
arXiv:2606. 07822v1 Announce Type: cross Abstract: As language models improve and become increasingly deployed to solve a variety of tasks, trustworthiness becomes essential.
By Nishant Subramani, Palash Goyal, Yiwen Song, Mani Malek, Yuan Xue, Tomas Pfister, Hamid Palangi
The paper introduces Chinese Decision, a family of Chinese-language models designed for single-pass probabilistic decision-making. The models generate logits for candidate choices in one forward pass and normalize them without autoregressive generation, supporting Boolean, categorical, and ordinal decisions. A 0.8B model trained on 13.048 million decisions achieves 72.24% general-domain accuracy and 76.98% average accuracy across medicine, law, and finance, while a 2B model shows comparable performance after partial training.
By Qiang Yan (Institute of Computing Technology, Chinese Academy of Sciences)
The paper critiques generative perplexity (gen‑PPL) as an unreliable metric for evaluating unconditional text generation, showing that it can be minimized by trivial, non‑diverse outputs. The authors demonstrate this by constructing simple samplers that achieve low gen‑PPL yet produce incoherent text, and they propose alternative evaluation tools—including distributional discrepancies, diversity diagnostics, and a compression‑based KL divergence measure—to better assess generative language models. They argue that future progress in diffusion language models should rely on a suite of metrics that capture diverse aspects of generated text quality.
By Antonio Franca, Alexander Tong