Pinocchio is an external calibrator that provides fast uncertainty estimates for black-box language models. It predicts the correctness of responses from seven trained LLMs with an AUROC of 0.862 and can transfer zero‑shot to thirteen unseen models from eight organizations. The method requires only a single forward pass and no access to the target model’s internal states, and a lightweight 0.8B checkpoint achieves comparable performance.
By Kevin David Hayes, Arka Pal, Haosong Zhang, Tom Goldstein, Micah Goldblum
arXiv:2601. 16398v3 Announce Type: replace-cross Abstract: Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators.
By Hannah Cyberey, Yangfeng Ji, David Evans
arXiv:2604. 22167v2 Announce Type: replace-cross Abstract: Language models are increasingly capable and are being rapidly deployed on a population-level scale.
By Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu, Rajesh Ranganath, Kathleen McKeown, He He
The paper introduces Word-level Probability MIA (WPMIA), a black-box membership inference attack that estimates word-level generation probabilities via Monte Carlo sampling and local kernel smoothing, then aggregates them into a sequence-level likelihood estimator. By conditioning on different prefixes, WPMIA amplifies distributional differences between member and non-member texts, outperforming existing black-box baselines on open-source LLMs and achieving an average TPR@5%FPR of 42.0 on proprietary models such as GPT‑5‑Chat, Gemini‑2.5‑Flash, and Claude‑4.5‑Haiku.
By Shengjie Niu, Yeheng Ge, Jian Huang
arXiv:2608.30731v1 Announce Type: cross
Abstract: Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challen...
By Pratuat Amatya, Venktesh Viswanathan, Vinay Setty
arXiv:2511. 21140v4 Announce Type: replace Abstract: Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators.
By Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, Kangwook Lee
The paper introduces LoRD, a lightweight post‑hoc calibration framework designed to improve confidence reliability in language‑model‑based log anomaly detectors. LoRD learns route‑specific reliability models from latent representations of correctly classified validation samples and uses reconstruction distances to estimate prediction reliability. By selectively recalibrating high‑risk predictions, LoRD reduces overconfident errors while maintaining strong anomaly detection performance across four large‑scale log benchmark datasets.
By Bin Li, Dongdong Wang, Siyang Lu
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu
arXiv:2601.03087v2 Announce Type: replace
Abstract: Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM...
By David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gie{\ss}ing, Bettina Berendt, Pieter Delobelle
The paper presents a simple detector for model extraction attacks on large language model APIs. It frames detection as a benign‑calibrated traffic‑window distribution test, embedding queries into a semantic space and using maximum mean discrepancy (MMD) to compare against historical benign traffic. Evaluated on fourteen attacker‑normal query pairs across four extraction scenarios, MMD achieves near‑perfect true‑positive rates while maintaining a 0.3% false‑positive rate, outperforming several existing baselines.
By Shuze Liu, Qianwen Guo, Yushun Dong
arXiv:2603. 24929v2 Announce Type: replace Abstract: Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment.
By Farhan Ahmed, Yuya Jeremy Ong, Chad DeLuca
The paper argues that calibration—how well a language model’s confidence aligns with its actual correctness—should be a standard evaluation metric for large language models (LLMs). It notes that while calibration metrics exist, they are rarely applied outside specialized NLP subfields, leading to unverified confidence scores in new models, datasets, and benchmarks. The authors highlight the risks of miscalibration both at deployment (overconfident errors causing harm) and in research workflows (affecting LLM-as-a-judge, synthetic data generation, and active learning). They call for every NLP subfield to pair its primary performance metric with a calibration score, treating calibration as an essential property of every model.
By Mario Sanz-Guerrero, Katharina von der Wense