arXiv AI
POOL: Propagated Uncertainty Over Lookalikes
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 12
Attention-Path Fragility as an Uncertainty Signal in Large Language Models
arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.
arXiv AI
Jul 29
Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs
arXiv:2607. 25600v1 Announce Type: cross Abstract: Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation.
Hugging Face Trending Papers
Sep 29
Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertaint...
arXiv AI
Sep 30
Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
The paper introduces Divergent Token Confidence (DTC), a method that estimates large language model confidence by counting tokens where two models strongly disagree during decoding. DTC uses Jensen-Shannon divergence between next-token distributions along the same reasoning trajectory and shows a near-negative correlation with answer accuracy. Experiments on multiple model families and six mathematical benchmarks demonstrate that DTC improves calibration over traditional probability-based and verbalized baselines, achieving lower expected calibration errors in both white-box and black-box settings.
arXiv AI
Aug 24
DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning
DirEAG introduces a Dirichlet Evidence Aggregation technique to calibrate verbalized confidence in large language models performing mathematical reasoning. By converting each elicited answer-confidence pair into calibrated soft evidence over candidate answers and a null state, it addresses prompt- and task-dependent bias that simple averaging or heuristic aggregation cannot handle. Experiments on GSM8K, SVAMP, and GSM-Hard with Qwen, Mistral, and Gemma models demonstrate that DirEAG achieves better calibration while maintaining competitive answer selection compared to existing methods.
arXiv AI
Sep 10
CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoning
CUSP (Collective Uncertainty through Semantic Opinion Pooling) is a training‑free framework that aggregates responses from multiple vision‑language models into a shared semantic space, producing a pooled opinion and two system‑level uncertainty signals: collective uncertainty (dispersion) and Jensen‑Shannon divergence (model conflict). It decomposes collective entropy into the mean of individual semantic entropies plus JSD, enabling reliable uncertainty estimation without token logits or calibration labels. In static ensembles, collective uncertainty outperforms baseline methods for error detection and abstention, while JSD excels in commercial settings, and the pooled prediction consistently improves accuracy over individual models. "whyItMatters":"CUSP provides a practical, model‑agnostic way to quantify system‑level reliability and improve decision‑making in multimodal reasoning tasks."