arXiv Machine Learning August 4, 2026 By Yuqicheng Zhu, Jialin Yu, Lin Li, Gengyuan Zhang, Zhen Yang, Steffen Staab, Puneet Dokania, Philip Torr, Jie Tang, Evgeny Kharlamov

Conformalized Large Language Models under Configuration Shift

Read the original on arXiv Machine Learning →

arXiv:2608. 01460v1 Announce Type: new Abstract: Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

Domain-Shift-Aware Conformal Prediction for Large Language Models

arXiv:2510. 05566v2 Announce Type: replace-cross Abstract: Large language models have achieved impressive performance across diverse tasks.

By Zhexiao Lin, Yuanyuan Li, Neeraj Sarna, Yuanyuan Gao, Michael von Gablenz

llms benchmarks safety

arXiv AI

Aug 6

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

arXiv:2608. 05064v1 Announce Type: cross Abstract: Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human.

By Jianru Shen

llms

arXiv AI

Jun 2

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

arXiv:2606. 01850v1 Announce Type: new Abstract: Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation.

By Yujia Tong, Yuxi Wang, Yunyang Wan, Tian Zhang, Junhao Dong, Jingling Yuan

llms nlp efficiency benchmarks safety

arXiv Machine Learning

Jun 30

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

arXiv:2606. 29054v1 Announce Type: new Abstract: Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.

By Varun Kotte

llms

arXiv AI

Jul 28

Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models

arXiv:2607. 24562v1 Announce Type: new Abstract: Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style.

By Murilo Salem, Lu\'isa B\"ohm, Daniel Pontes, Anderson Ferrugem

llms efficiency benchmarks

arXiv AI

Aug 7

Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning

arXiv:2604. 01170v2 Announce Type: replace-cross Abstract: While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs.

By Cai Zhou, Zekai Wang, Menghua Wu, Qianyu Julie Zhu, Flora C. Shi, Chenyu Wang, Ashia Wilson, Tommi Jaakkola, Stephen Bates

llms benchmarks

Related stories

Domain-Shift-Aware Conformal Prediction for Large Language Models

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models

Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning