arXiv Statistics ML

A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI

arXiv Machine Learning
Sep 14

A Ranking Approach for Measuring Calibration

The paper introduces rankECE, a new metric for assessing calibration error in predictive models. Unlike the widely used Expected Calibration Error (ECE), rankECE compares predictions with neighboring probability values, offering theoretical guarantees and empirical evidence that it better approximates ECE than traditional binned methods.

By Anirban Chatterjee, Rina Foygel Barber
arXiv Machine Learning
Jul 9

Optimal Conformal Prediction under Epistemic Uncertainty

arXiv:2505. 19033v2 Announce Type: replace-cross Abstract: Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees.

By Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies, Willem Waegeman, Aleksandar Bojchevski, Eyke H\"ullermeier
arXiv Machine Learning
Jun 3

Set-Preserving Calibration from Conformal P-Values to E-Values

arXiv:2606. 03600v1 Announce Type: cross Abstract: Standard conformal prediction (CP) procedures are typically formulated in terms of p-values, but reliance on p-values alone limits flexibility, for example, when combining dependent evidence across models or data splits.

By Nabil Alami, Jad Zakharia, Souhaib Ben Taieb
arXiv Machine Learning
Jul 27

Simulation-Based Empirical Bayes

arXiv:2607. 21843v1 Announce Type: cross Abstract: Empirical Bayes (EB) performs simultaneous inference across many related latent variables.

By Xinwei Shen, Diana Cai, Cheng Zhang, David M. Blei
arXiv Machine Learning
Sep 11

Conformal Calibration Transfer

Conformal Calibration Transfer addresses the challenge of applying conformal prediction when labeled calibration data is only available in a source space, while predictions are needed in a target space linked via unlabeled paired observations. The proposed Transported Conformal Calibration (TCC) method transports source calibration into the target domain and then corrects residual mismatches using only unlabeled target inputs, with two variants: TCC‑KS, which conservatively adjusts calibration based on a label‑free uncertainty surrogate, and weighted‑TCC, which reweights transported calibration for efficiency when weights are stable. Finite‑sample target‑domain coverage guarantees are provided, and experiments on CIFAR‑100‑C, Tiny‑ImageNet‑C, and SEN12MS demonstrate reliable coverage transfer without labeled target data, along with label‑free diagnostics to signal when correction is required.

By Achref Doula