arXiv Machine Learning August 4, 2026 By Yuqicheng Zhu, Jialin Yu, Lin Li, Gengyuan Zhang, Zhen Yang, Steffen Staab, Puneet Dokania, Philip Torr, Jie Tang, Evgeny Kharlamov

Conformalized Large Language Models under Configuration Shift

Read the original on arXiv Machine Learning →

arXiv:2608. 01460v1 Announce Type: new Abstract: Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 2

Domain-Shift-Aware Conformal Prediction for Large Language Models

arXiv:2510. 05566v2 Announce Type: replace-cross Abstract: Large language models have achieved impressive performance across diverse tasks.

By Zhexiao Lin, Yuanyuan Li, Neeraj Sarna, Yuanyuan Gao, Michael von Gablenz

llms benchmarks safety

arXiv AI

Aug 6

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

arXiv:2608. 05064v1 Announce Type: cross Abstract: Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human.

By Jianru Shen

llms

arXiv AI

Jun 2

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

arXiv:2606. 01850v1 Announce Type: new Abstract: Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation.

By Yujia Tong, Yuxi Wang, Yunyang Wan, Tian Zhang, Junhao Dong, Jingling Yuan

llms nlp efficiency benchmarks safety

arXiv Machine Learning

Jun 30

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

arXiv:2606. 29054v1 Announce Type: new Abstract: Large language models (LLMs) deployed for structured generation (NER, JSON extraction, QA, and classification) lack formal reliability guarantees, and standard heuristic abstention policies miss user-specified risk targets by 7.

By Varun Kotte

llms

arXiv AI

Aug 24

Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models

arXiv:2608.21019v1 Announce Type: cross Abstract: Quantization is widely used to deploy large language models, but its effect on uncertainty behavior, such as confidence, margins, and abstention, is...

By Zhen Yang, Sizai Hou, Kaiwen Zheng, Yaofang Liu, Liang He, Yixuan Chen, Kangning Cui

llms nlp efficiency benchmarks

arXiv Computation and Language

Sep 22

Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

arXiv:2604.08974v2 Announce Type: replace Abstract: Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection an...

By Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung

llms nlp fine-tuning safety

One email a morning, machine-written

Related stories

Domain-Shift-Aware Conformal Prediction for Large Language Models

Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation

Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models

Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning