arXiv AI September 23, 2026 By Kevin David Hayes, Arka Pal, Haosong Zhang, Tom Goldstein, Micah Goldblum

Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models

Pinocchio is an external calibrator that provides fast uncertainty estimates for black-box language models. It predicts the correctness of responses from seven trained LLMs with an AUROC of 0.862 and can transfer zero‑shot to thirteen unseen models from eight organizations. The method requires only a single forward pass and no access to the target model’s internal states, and a lightweight 0.8B checkpoint achieves comparable performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 19

A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models

arXiv:2606. 19868v1 Announce Type: new Abstract: Although large language models (LLMs) have shown strong capabilities across a wide range of tasks, their outputs often remain unreliable and may contain hallucinations, making uncertainty estimation (UE) essential for building trustworthy LLMs.

By Jiayi Wang, Xu-Yao Zhang

llms agents benchmarks safety

arXiv AI

Aug 5

LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics

arXiv:2603. 24929v2 Announce Type: replace Abstract: Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment.

By Farhan Ahmed, Yuya Jeremy Ong, Chad DeLuca

llms safety

arXiv Machine Learning

Sep 7

Single-Query Black-Box Calibration Auditing via Logit Bias

The paper introduces a method for auditing the calibration of large language models (LLMs) that only exposes a logit_bias parameter. By mathematically manipulating this parameter, the authors can evaluate exact probability thresholds with a single query per sample, enabling a provably consistent estimator of True Calibration Error for binary tasks. This approach offers an efficient framework for auditing black‑box foundation models despite limited access to continuous output probabilities.

By Roman Plaud, Antoine Saillenfest, Matthieu Labeau, Thomas Bonald, Willem Waegeman

llms safety

arXiv Computation and Language

Sep 1

A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantification

arXiv:2604.11662v2 Announce Type: replace Abstract: Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation, motivating a growing inter...

By Joe Stacey, Hadas Orgad, Kentaro Inui, Benjamin Heinzerling, Nafise Sadat Moosavi

llms

arXiv AI

Aug 11

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.

By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

llms agents benchmarks safety

arXiv AI

Jun 12

Cross-Model Disagreement as a Label-Free Correctness Signal

arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.

By Matt Gorbett, Suman Jana

llms benchmarks

One email a morning, machine-written

Related stories

A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models

LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics

Single-Query Black-Box Calibration Auditing via Logit Bias

A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantification

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

Cross-Model Disagreement as a Label-Free Correctness Signal