arXiv AI
Jun 2
Domain-Shift-Aware Conformal Prediction for Large Language Models
arXiv:2510. 05566v2 Announce Type: replace-cross Abstract: Large language models have achieved impressive performance across diverse tasks.
Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift
The paper investigates how changes in inference-time settings—such as quantization and temperature—affect conformal prediction (CP) for language models. It introduces the concept of inference-time model shift, showing that such shifts break split-conformal exchangeability even when the underlying model and data distribution remain unchanged. Experiments on eight language models, four benchmarks, and twelve inference configurations reveal that matched evaluation protocols can significantly overestimate deployment-time coverage, with effects varying across models and datasets.
Related stories
arXiv Machine Learning
Aug 4
Conformalized Large Language Models under Configuration Shift
arXiv:2608. 01460v1 Announce Type: new Abstract: Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability.
arXiv AI
Jun 18
DRIFT: Refining Instruction Data via On-Policy Data Attribution
arXiv:2606. 18307v1 Announce Type: cross Abstract: Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs).
arXiv AI
Aug 7
Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
arXiv:2604. 01170v2 Announce Type: replace-cross Abstract: While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs.
arXiv AI
Jun 9
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
arXiv AI
Oct 15
Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings
The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.
arXiv AI
Jul 15
Scaling Point-in-Time Language Models
arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.
arXiv Machine Learning
Jul 14
Domain-Aware Scaling Laws Uncover Data Synergy
arXiv:2607. 11052v1 Announce Type: new Abstract: Machine learning progress is often attributed to scaling model size and dataset volume, yet the composition of data can be just as consequential.
arXiv Machine Learning
Jun 11
Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation
arXiv:2606. 11865v1 Announce Type: cross Abstract: Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient.
arXiv Machine Learning
Sep 24
Rolling Conformal Prediction in Sequential Model Training
Rolling Conformal Prediction (rolling‑CP) is a distribution‑free predictive inference method designed for sequential model training. It calibrates each incoming observation against the current predictor and incorporates it into future training, eliminating the need for data splitting. For exchangeable data, rolling‑CP guarantees marginal coverage with a universal factor‑two bound, and for i.i.d. streams it provides high‑probability training‑conditional validity over time, improving to the target level under stability conditions.
arXiv AI
Sep 30
OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
The paper introduces OTROPE, a likelihood‑free method for off‑policy evaluation of large language models (LLMs) that uses optimal transport to align labeled samples from a behavior model with unlabeled samples from a target model in a semantic space. OTROPE corrects human‑labeled residuals with proxy predictors, achieving a doubly robust evaluation without requiring behavior‑policy modeling or density‑ratio estimation. The authors provide theoretical guarantees for consistency and convergence, and demonstrate through synthetic and real LLM tasks that OTROPE outperforms existing baselines and can elevate weaker evaluators to match or exceed stronger ones.
arXiv Machine Learning
Sep 22
SupportCal: Label-Free Calibration of Post-Trained LLMs via Reference Support and Corroboration
arXiv:2609.24303v1 Announce Type: new Abstract: Post-training often improves task performance but can degrade confidence calibration, leaving post-trained language models (PoLMs) more overconfident t...