Pinocchio is an external calibrator that provides fast uncertainty estimates for black-box language models. It predicts the correctness of responses from seven trained LLMs with an AUROC of 0.862 and can transfer zero‑shot to thirteen unseen models from eight organizations. The method requires only a single forward pass and no access to the target model’s internal states, and a lightweight 0.8B checkpoint achieves comparable performance.
arXiv:2606. 19868v1 Announce Type: new Abstract: Although large language models (LLMs) have shown strong capabilities across a wide range of tasks, their outputs often remain unreliable and may contain hallucinations, making uncertainty estimation (UE) essential for building trustworthy LLMs.
arXiv:2603. 24929v2 Announce Type: replace Abstract: Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment.
The paper introduces a method for auditing the calibration of large language models (LLMs) that only exposes a logit_bias parameter. By mathematically manipulating this parameter, the authors can evaluate exact probability thresholds with a single query per sample, enabling a provably consistent estimator of True Calibration Error for binary tasks. This approach offers an efficient framework for auditing black‑box foundation models despite limited access to continuous output probabilities.
arXiv:2604.11662v2 Announce Type: replace Abstract: Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation, motivating a growing inter...
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.
arXiv:2608. 19323v1 Announce Type: cross Abstract: Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs).
arXiv:2608. 02879v1 Announce Type: new Abstract: The widespread adoption of proprietary Large Language Models (LLMs) accessed strictly through closed APIs has created a critical challenge for responsible deployment: a fundamental lack of interpretability.
arXiv:2609.10122v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance across a broad range of classification settings, yet the reliability of their predictions...
arXiv:2608.30731v1 Announce Type: cross Abstract: Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challen...
arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.