The paper introduces the Fragility Index (FI), a new performance metric that measures the tail risk of confident misclassifications in classification models. FI is framed within a robust satisficing framework to assess classifier sensitivity under distributional uncertainty, and the authors develop a surrogate-based training method with provable bounds linking training fragility to ranking-error FI. The approach is extended to common loss functions and includes an FI-based regularizer for deep neural networks, with empirical studies on medical diagnosis tasks showing that FI reveals risk not captured by accuracy or AUC and leads to models with lower operational costs in a human–AI diagnostic workflow.
By Zheng Cui, Daniel Zhuoyu Long, Jin Qi, Chen Yang, Ruohan Zhan
The paper introduces a method for learning from multiple experts who provide interval labels, addressing both within‑label imprecision and between‑expert variation. It harmonizes diverse label vocabularies into a shared probabilistic space, retains individual intervals using a mixture of Beta distributions, and decomposes predictive uncertainty into components that are matched to their corresponding sources of label uncertainty. On sea‑ice concentration data, the approach achieves a 31% reduction in mean absolute error compared to hard‑label baselines and outperforms several aggregation and interval‑regression methods.
By Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett, Farnoush Banaei-Kashani
The paper investigates how pre‑training strategy, dataset size, and domain affect uncertainty estimation in vision medical foundation models. It compares point‑prediction calibration with conformal (region) prediction across retinal, histopathological, and chest X‑ray models, finding that domain‑specific, self‑supervised pre‑training yields better calibration and more efficient conformal sets. The study shows that standard recalibration alone cannot fully reconcile uncertainty differences between models trained on different data sources.
By Haoxu Huang, Narges Razavian
arXiv:2606. 24959v1 Announce Type: new Abstract: Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors.
By Stefan Haas, Luca Killmaier, Alireza Javanmardi, Eyke H\"ullermeier
arXiv:2604.22391v2 Announce Type: replace-cross
Abstract: The Super Learner (SL) is a widely used ensemble method that combines point predictions from a library of learners based on their predictive...
By Zhanli Wu, Fabrizio Leisen, Miguel-Angel Luque-Fernandez, F. Javier Rubio
arXiv:2609.25388v1 Announce Type: cross
Abstract: A classical question in statistics is which observable quantities to condition on when drawing inferences about unobservable targets. For conformal p...
By Xuelin Yang, Baihe Huang, Yilong Hou, Guido Imbens, Michael I. Jordan
The paper introduces a new metric called Localized AUC (LAUC), a weighted version of the traditional AUC, to detect performance heterogeneity in risk prediction models. Using LAUC, the authors develop a method named LOCATOR that identifies patient cohorts where the model’s predictions are inaccurate. They validate LOCATOR on both real-world data and simulations and provide a statistical foundation for applying LAUC in this context.
By Wenye Ren, Anna Gy\H{o}rffy-Kerekes, Tancredi Dorsi, Olga Demler
arXiv:2607. 16675v1 Announce Type: cross Abstract: A point prediction that is well calibrated on average can still be systematically biased conditional on its own value, undermining its use in downstream decision-making.
By Daniel Bensimon, Sean Xiang Yu, Eric D. Kolaczyk, Archer Y. Yang
arXiv:2601. 21455v2 Announce Type: replace-cross Abstract: Conformal prediction(CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length.
By Yizhou Min, Yizhou Lu, Lanqi Li, Zhen Zhang, Jiaye Teng
arXiv:2610.09392v1 Announce Type: new
Abstract: Reliable volumetric segmentation is critical for clinical diagnostics, yet foundation models such as MedSAM remain deterministic and lack calibrated un...
By Shadi Alijani, Fereshteh Aghaee Meibodi, Homayoun Najjaran
arXiv:2607. 15394v1 Announce Type: new Abstract: Black-box models limit the adoption of artificial intelligence in medicine due to their lack of interpretability and reproducibility.
By Antony Garcia, Adrian Noriega, Gabrielle Britton, Xinming Huang
arXiv:2607. 22984v1 Announce Type: cross Abstract: Machine learning models for clinical prediction tasks, such as in-hospital mortality and sepsis onset, routinely achieve high AUROC scores.
By Jie JW Wu, Feiyu E, Bo Chen