arXiv Machine Learning

An analysis of binary isotonic regression: degrees of freedom and implications for calibration

arXiv:2607. 27301v1 Announce Type: cross Abstract: Isotonic regression is a canonical tool for estimating monotone functions and calibrating probabilistic predictors.

arXiv Machine Learning
Sep 14

A Ranking Approach for Measuring Calibration

The paper introduces rankECE, a new metric for assessing calibration error in predictive models. Unlike the widely used Expected Calibration Error (ECE), rankECE compares predictions with neighboring probability values, offering theoretical guarantees and empirical evidence that it better approximates ECE than traditional binned methods.

By Anirban Chatterjee, Rina Foygel Barber
arXiv Machine Learning
Jul 21

Isotonic Conformal Prediction

arXiv:2607. 16675v1 Announce Type: cross Abstract: A point prediction that is well calibrated on average can still be systematically biased conditional on its own value, undermining its use in downstream decision-making.

By Daniel Bensimon, Sean Xiang Yu, Eric D. Kolaczyk, Archer Y. Yang
arXiv Machine Learning
Aug 24

Truthful Calibration Measures for Sequential Prediction

The paper investigates the feasibility of exact truthfulness in calibration measures for sequential binary prediction. It proves that exact truthfulness cannot coexist with completeness and soundness, even when outcomes are independent. The authors then provide two reductions that transform any base calibration measure into additively or multiplicatively approximately truthful ones, achieving a multiplicative truthfulness guarantee that improves upon previous results.

By Anagha Gokul, Jason Hartline, Lunjia Hu, Jonathan Ullman, Yifan Wu