arXiv Machine Learning
Calibration Count Reuse: Validity Does Not Determine Efficiency
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 31
Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
arXiv:2606. 03328v3 Announce Type: replace Abstract: Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects.
arXiv Machine Learning
Aug 24
The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
arXiv:2608.21262v1 Announce Type: cross Abstract: Many machine-learning systems set a threshold at a quantile of a calibration set: conformal predictors that promise 90% coverage by drawing their cut...
arXiv Machine Learning
Aug 11
ARC: Augmented-Rank Conformalization for Changepoint Localization --- Finite-Sample Validity and Distribution-Robust Efficiency
arXiv:2608. 08424v1 Announce Type: cross Abstract: Conformal changepoint localization turns any score into a confidence set for the changepoint with finite-sample coverage.
arXiv AI
Oct 16
Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage
The paper argues that post‑training pruning of large language models should use a calibration set that covers multiple capability domains rather than a single source. A balanced four‑way mixture of sources consistently outperforms any single source across sparsity levels, model scales, and held‑out benchmarks. When aligned corpora are unavailable, the IGSP method approximates this mixture using only the base model and an evaluation taxonomy, achieving significant gains over prior self‑generation approaches.
arXiv AI
Oct 1
How Much Can Reliability Drift Under a Fixed Confidence Distribution?
arXiv:2609.38917v1 Announce Type: new Abstract: A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliab...
arXiv Machine Learning
Sep 29
Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift
The paper introduces a missingness‑aware conformal calibration method for mortality prediction that accounts for cross‑hospital distribution shifts. By selecting a measurement on an independent sample, grouping patients by whether that measurement is recorded, and applying Mondrian calibration within each group, the method avoids reusing calibration outcomes. Experiments on eICU and MIMIC‑IV data show that, compared to pooled calibration, it reduces the worst‑group coverage gap by a median of 1.9 percentage points across six settings, though the benefit varies with predictor and hospital.