arXiv AI
Aug 21
Improved Confidence Estimates for Black-Box Large Language Models
arXiv:2608. 19323v1 Announce Type: cross Abstract: Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs).
ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification
Related stories
arXiv Machine Learning
Jun 25
Don't Go Breaking My LLM: The Impact of Pruning Attention Layers on Explanation Faithfulness and Confidence Calibration
arXiv:2606. 24970v1 Announce Type: new Abstract: Pruning Large Language Models (LLMs) reduces memory and inference costs by removing parts of the network, producing smaller models that retain most of their accuracy.
arXiv Machine Learning
Jul 13
Accelerating Large Language Model Inference with Self-Supervised Early Exits
arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.
arXiv AI
Jul 7
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.
arXiv Computation and Language
Aug 28
The Score Granularity Gap in Black-Box LLM Classification: A Comparative Study of Confidence Constructions
The paper investigates the "score granularity gap" in black-box large language model (LLM) classifiers, asking how finely a confidence score can be thresholded for deployment. By comparing seven confidence construction methods across 25 model-dataset pairs, the authors find that single-shot verbalized confidence, when properly converted to a probability, ranks well but offers only a few distinct threshold values, limiting operational flexibility. The study also shows that multi-query aggregation can improve weak models but may harm strong ones, and provides concrete guidance for deployment trade-offs.
arXiv AI
Jul 9
Future Confidence Distillation in Large Language Models
arXiv:2607. 07626v1 Announce Type: cross Abstract: Reliable confidence estimation is essential for deploying large language models (LLMs) in confidence-aware systems, where downstream decisions such as retrieval, tool use, and adaptive computation depend on accurately estimating answer reliability.
arXiv Computation and Language
6d ago
A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs
The paper introduces a Calibrated Reflection approach to improve confidence estimation in Large Language Models (LLMs). It combines structured reasoning with a distance‑aware calibration technique, featuring a Maximum Confidence Selection method, a reflection‑based prompting mechanism, and an ordinal‑aware calibration strategy. Experiments on datasets such as HelpSteer2, Llama T‑REx, and a proprietary conversational set show the method works for both conversational and fact‑based classification tasks.
Hugging Face Trending Papers
Aug 5
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs.
Hugging Face Trending Papers
Jul 7
SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation
Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors.
arXiv AI
Jun 29
From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models
arXiv:2606. 27679v1 Announce Type: cross Abstract: Probe-based uncertainty estimation (UE) has emerged as a prominent approach to detect hallucinations in Large Language Models (LLMs) by learning uncertainty from internal model signals.
arXiv AI
Aug 6
Calibrating Transformer Attention via Task-Space Sensitivity Feedback
arXiv:2512. 20661v2 Announce Type: replace Abstract: Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens.
arXiv AI
Sep 1
Training-free LLM Verification via Recycling Few-shot Examples
arXiv:2506.17251v3 Announce Type: replace-cross Abstract: Although large language models (LLMs) have achieved remarkable performance, the inherent stochasticity of their reasoning processes and varyi...