arXiv Machine Learning
Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
2d ago
Fragility-aware Classification for Risk Characterization and Robust Learning
The paper introduces the Fragility Index (FI), a new performance metric that measures the tail risk of confident misclassifications in classification models. FI is framed within a robust satisficing framework to assess classifier sensitivity under distributional uncertainty, and the authors develop a surrogate-based training method with provable bounds linking training fragility to ranking-error FI. The approach is extended to common loss functions and includes an FI-based regularizer for deep neural networks, with empirical studies on medical diagnosis tasks showing that FI reveals risk not captured by accuracy or AUC and leads to models with lower operational costs in a human–AI diagnostic workflow.
arXiv Machine Learning
Oct 3
Uncertainty-Aware Learning from Multi-Expert Interval Targets
The paper introduces a method for learning from multiple experts who provide interval labels, addressing both within‑label imprecision and between‑expert variation. It harmonizes diverse label vocabularies into a shared probabilistic space, retains individual intervals using a mixture of Beta distributions, and decomposes predictive uncertainty into components that are matched to their corresponding sources of label uncertainty. On sea‑ice concentration data, the approach achieves a 31% reduction in mean absolute error compared to hard‑label baselines and outperforms several aggregation and interval‑regression methods.
arXiv Machine Learning
Sep 1
Uncertainty of Vision Medical Foundation Models
The paper investigates how pre‑training strategy, dataset size, and domain affect uncertainty estimation in vision medical foundation models. It compares point‑prediction calibration with conformal (region) prediction across retinal, histopathological, and chest X‑ray models, finding that domain‑specific, self‑supervised pre‑training yields better calibration and more efficient conformal sets. The study shows that standard recalibration alone cannot fully reconcile uncertainty differences between models trained on different data sources.
arXiv Machine Learning
Jun 25
Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score
arXiv:2606. 24959v1 Announce Type: new Abstract: Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors.
arXiv Machine Learning
Sep 10
Conformalized Super Learner
arXiv:2604.22391v2 Announce Type: replace-cross Abstract: The Super Learner (SL) is a widely used ensemble method that combines point predictions from a library of learners based on their predictive...
arXiv Machine Learning
Sep 23
PICPIs: Prediction-Interval-Conditional Prediction Intervals
arXiv:2609.25388v1 Announce Type: cross Abstract: A classical question in statistics is which observable quantities to condition on when drawing inferences about unobservable targets. For conformal p...