arXiv AI
Jul 7
The Anatomy of Uncertainty in LLMs
arXiv:2603. 24967v2 Announce Type: replace Abstract: Understanding why a large language model (LLM) is uncertain about the response is important for their reliable deployment.
LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics
arXiv:2603. 24929v2 Announce Type: replace Abstract: Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment.
Related stories
arXiv Machine Learning
Jun 29
On the Effect of Uncertainty on Layer-wise Inference Dynamics
arXiv:2507. 06722v2 Announce Type: replace-cross Abstract: Understanding how large language models (LLMs) internally represent and process their predictions is central to detecting uncertainty and preventing hallucinations.
Hugging Face Trending Papers
Jun 22
The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models
Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.
arXiv AI
Jun 12
Cross-Model Disagreement as a Label-Free Correctness Signal
arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.
arXiv AI
Sep 23
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator that provides fast uncertainty estimates for black-box language models. It predicts the correctness of responses from seven trained LLMs with an AUROC of 0.862 and can transfer zero‑shot to thirteen unseen models from eight organizations. The method requires only a single forward pass and no access to the target model’s internal states, and a lightweight 0.8B checkpoint achieves comparable performance.
arXiv Machine Learning
Jun 19
Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence
arXiv:2606. 19353v1 Announce Type: cross Abstract: In-Context Learning (ICL) allows LLMs to adapt to new tasks from a few demonstrations, but its reliability remains a concern: predictions are highly sensitive to both prompt design and the model's ability to understand the context, obscuring whether failures arise from data properties or model limitations.
arXiv AI
Sep 3
From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs
The paper investigates hallucination detection in black‑box large language models by leveraging two accessible signals: semantic entropy, which captures disagreement among sampled response meanings, and token‑level uncertainty derived from log‑probabilities. It introduces a TopK aggregation technique, a hybrid CoCoA method combining uncertainty with semantic dissimilarity, and two supervised approaches—Gated and Stacked—that integrate token and semantic features. Across seven benchmarks and four language models, the supervised Stacked method performs best in many cases, while TopK and CoCoA remain competitive without labeled data, though all methods require careful threshold calibration.
arXiv Machine Learning
Aug 27
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
The paper introduces Functional Entropy, a new uncertainty quantification technique for assessing the functional correctness of code generated by large language models. It evaluates token‑probability and sampling‑based methods across three programming languages and five LLMs, finding that token‑probability approaches generalize well while NLI‑based sampling fails due to semantic clustering. Functional equivalence methods, which replace NLI with an LLM‑based functional assessment, achieve superior AUROC and calibration in most model‑benchmark combinations.
arXiv AI
Aug 26
Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review
arXiv:2504.18346v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination, i.e., confidently outputting incorrect inf...
arXiv AI
Jul 7
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
arXiv AI
Jul 3
TokenScope: Token-Level Explainability and Interpretability for Code-Oriented Tasks in Large Language Models
arXiv:2607. 01235v1 Announce Type: cross Abstract: Understanding how Large Language Models (LLMs) make token-level decisions during code generation remains a major challenge for both researchers and practitioners.
arXiv AI
Jun 19
A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models
arXiv:2606. 19868v1 Announce Type: new Abstract: Although large language models (LLMs) have shown strong capabilities across a wide range of tasks, their outputs often remain unreliable and may contain hallucinations, making uncertainty estimation (UE) essential for building trustworthy LLMs.