← Back to all news
Hugging Face Trending Papers

Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets

The paper introduces a local audit method that identifies where a prediction model is miscalibrated and for whom, by grouping similar observations into ‘pockets’ using random‑forest similarity and ridge‑regression. The audit, which requires only features, outcomes, and fixed predictions, demonstrates superior detection of group‑specific errors in simulations and real synthetic data, revealing nuanced over‑ and under‑prediction patterns that overall metrics like AUC can miss.

Related stories

arXiv Computation and Language
Sep 11

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

The study audited ten different classifiers—including linear, tree‑ensemble, neural, glass‑box, and tabular foundation models—on national health survey data to predict myocardial infarction. By systematically removing features that could cause target leakage, the authors found that all models’ AUROC scores collapsed into a narrow band, indicating that reported high accuracy in prior work was largely due to leakage rather than model sophistication. The glass‑box explainable boosting machine performed comparably to other models while being much faster, and the authors demonstrated that fairness, calibration, and uncertainty can be audited and repaired without sacrificing performance.

By Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz
safety
More like this →
arXiv Machine Learning
Sep 29

Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

The paper introduces a missingness‑aware conformal calibration method for mortality prediction that accounts for cross‑hospital distribution shifts. By selecting a measurement on an independent sample, grouping patients by whether that measurement is recorded, and applying Mondrian calibration within each group, the method avoids reusing calibration outcomes. Experiments on eICU and MIMIC‑IV data show that, compared to pooled calibration, it reduces the worst‑group coverage gap by a median of 1.9 percentage points across six settings, though the benefit varies with predictor and hospital.

By Liang You, Dongwen Ou, Hengyu Shi, Siyuan Dai
More like this →
arXiv Machine Learning
Oct 19

Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction

The paper introduces alert‑list concordance, a metric that evaluates whether different imputation strategies for ICU sepsis prediction models identify the same high‑risk patients, measured by Jaccard similarity of top‑k% alert sets with bootstrap confidence intervals. Using PhysioNet/CinC 2019 data, the authors compare nine imputation methods and a native‑missingness baseline, finding that many methods have similar AUROC yet produce alert lists that overlap only 51–75% of the time. ForwardFill stands out as an outlier, achieving higher discrimination but lower overlap (40–45%).

By Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe Ma
More like this →
arXiv AI
Aug 6

Distributionally Robust Transfer Learning with Structurally Missing Covariates, with Application to Cross-National Cardiac Arrest Prediction

arXiv:2605. 24212v2 Announce Type: replace-cross Abstract: Deploying clinical prediction models across healthcare systems often fails when key training covariates are unavailable at deployment and labeled outcomes are limited in the target domain.

By Siqi Li, Chuan Hong, Ziye Tian, Benjamin Sieu-Hon Leong, Koshi Nakagawa, Hideharu Tanaka, Sang Do Shin, Khuong Quoc Dai, Do Ngoc Son, Marcus Eng Hock Ong, Nan Liu, Molei Liu
fine-tuningsafety
More like this →
arXiv Machine Learning
1d ago

Leakage- and Label-Aware Benchmarking of Class-Imbalance Strategies for Multimodal Chest X-Ray and EHR Classification

arXiv:2611.07950v1 Announce Type: new Abstract: Benchmarks of class-imbalance strategies for multimodal chest X-ray (CXR) and electronic health record (EHR) prediction are often evaluated on record-l...

By Dang-Khoa Bui, Quang-Huy Tran, Minh-Khoi Nguyen-Bui, Duc-Tuan Ngo, Thanh-Trong Tran, Tuan-Khoi Nguyen, Hoang-Anh Ngo
ragmultimodalbenchmarks
More like this →
arXiv Machine Learning
Oct 23

Local Discrimination of Binary Classifiers via Localized AUC Functional

The paper introduces a new metric called Localized AUC (LAUC), a weighted version of the traditional AUC, to detect performance heterogeneity in risk prediction models. Using LAUC, the authors develop a method named LOCATOR that identifies patient cohorts where the model’s predictions are inaccurate. They validate LOCATOR on both real-world data and simulations and provide a statistical foundation for applying LAUC in this context.

By Wenye Ren, Anna Gy\H{o}rffy-Kerekes, Tancredi Dorsi, Olga Demler
More like this →