← Back to all news
arXiv Machine Learning By Achref Doula, Max M\"uhlh\"auser

Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift

Read the original on arXiv Machine Learning →

The paper investigates how changes in inference-time settings—such as quantization and temperature—affect conformal prediction (CP) for language models. It introduces the concept of inference-time model shift, showing that such shifts break split-conformal exchangeability even when the underlying model and data distribution remain unchanged. Experiments on eight language models, four benchmarks, and twelve inference configurations reveal that matched evaluation protocols can significantly overestimate deployment-time coverage, with effects varying across models and datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 4

Conformalized Large Language Models under Configuration Shift

arXiv:2608. 01460v1 Announce Type: new Abstract: Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability.

By Yuqicheng Zhu, Jialin Yu, Lin Li, Gengyuan Zhang, Zhen Yang, Steffen Staab, Puneet Dokania, Philip Torr, Jie Tang, Evgeny Kharlamov
llmsefficiency
More like this →
arXiv AI
Oct 15

Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings

The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.

By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
llmsragfine-tuningbenchmarkssafety
More like this →