The paper introduces V-ECE, a variational estimator for general Expected Calibration Errors (ECE) that avoids binning and works with any convex divergence, including Lp distances. By using prediction‑dependent proper scores and a residual‑fitted temperature scaling via gradient boosting, V‑ECE provides a lower‑bound estimate of true calibration error. Experiments on semi‑synthetic benchmarks show V‑ECE outperforms existing binary and multiclass estimators, with additional theoretical insights on bias and confidence estimation.
arXiv:2605. 30188v2 Announce Type: replace-cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated.
arXiv:2408. 08998v4 Announce Type: replace-cross Abstract: Recent advances in machine learning have significantly improved prediction accuracy in various applications.
arXiv:2602. 13362v2 Announce Type: replace-cross Abstract: A key challenge in probabilistic regression is ensuring that predictive distributions accurately reflect true empirical uncertainty.
arXiv:2608. 10372v1 Announce Type: new Abstract: Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining.
The paper introduces rankECE, a new metric for assessing calibration error in predictive models. Unlike the widely used Expected Calibration Error (ECE), rankECE compares predictions with neighboring probability values, offering theoretical guarantees and empirical evidence that it better approximates ECE than traditional binned methods.
The paper introduces the Fragility Index (FI), a new performance metric that measures the tail risk of confident misclassifications in classification models. FI is framed within a robust satisficing framework to assess classifier sensitivity under distributional uncertainty, and the authors develop a surrogate-based training method with provable bounds linking training fragility to ranking-error FI. The approach is extended to common loss functions and includes an FI-based regularizer for deep neural networks, with empirical studies on medical diagnosis tasks showing that FI reveals risk not captured by accuracy or AUC and leads to models with lower operational costs in a human–AI diagnostic workflow.
The paper introduces soft‑label‑based estimators for the Bayes‑optimal balanced error rate (BER) and area under the ROC curve (AUC), extending from a clean setting with known class priors to a realistic scenario with unknown priors and corrupted soft labels. It also adapts the FeeBee evaluation framework to assess these estimators without needing the true optimum, providing practical evaluation scores for any estimator of optimal BER or AUC. Experiments on synthetic and real datasets confirm the effectiveness of both the estimators and the evaluation method.
arXiv:2606. 25188v1 Announce Type: new Abstract: Efficient uncertainty quantification (UQ) is essential for trustworthy large-scale learning.
arXiv:2608. 09768v1 Announce Type: new Abstract: A prediction that is both confident and wrong is a critical reliability failure because it can bypass abstention and human review precisely when the model is mistaken.
arXiv:2606. 28654v1 Announce Type: cross Abstract: Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods.
Reliable confidence estimates are essential in semantic segmentation, especially in safety-critical settings where overconfident errors can mislead downstream decisions. Yet modern segmentation models often remain miscalibrated.
arXiv:2502. 15131v4 Announce Type: replace-cross Abstract: We study the fundamental problem of calibrating a linear binary classifier of the form $\sigma(\hat{w}^\top x)$, where the feature vector $x$ is Gaussian, $\sigma$ is a link function, and $\hat{w}$ is an estimator of the true linear weight $w^\star$.