arXiv AI August 28, 2026 By Chorok Lee

Diagnosing Conformal Prediction Failures Under Distribution Shift: A COVID-19 Case Study

The paper introduces SHAP concentration as a pre‑deployment diagnostic for detecting when conformal prediction will fail under distribution shift, specifically in gradient‑boosted classifiers. Using a COVID‑19 supply‑chain case study, the authors show that higher feature‑importance concentration correlates with larger drops in coverage, while standard shift detectors cannot differentiate between catastrophic and robust outcomes. The diagnostic is validated on additional datasets, and a formal theorem links concentration to worsening conformity‑score bounds, though it does not capture global‑sensitivity failures in neural networks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Aug 31

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

arXiv:2608.27704v1 Announce Type: new Abstract: When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version,...

By Madhusudan Srinivasan, Namith Nishal Raphae

arXiv AI

Aug 18

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

arXiv:2505. 04608v5 Announce Type: replace-cross Abstract: Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but also continual, post-deployment monitoring to quickly detect and address any unsafe behavior.

By Drew Prinster, Xing Han, Anqi Liu, Suchi Saria

benchmarks

arXiv Machine Learning

Jun 16

Audited Conformal Prediction for Classification under Unknown Distribution Shift

arXiv:2606. 14909v1 Announce Type: cross Abstract: We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift.

By Yanfei Zhou, Rizal Fathony, Nam H. Nguyen, Matteo Sesia

arXiv AI

Aug 17

Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

arXiv:2608. 14089v1 Announce Type: new Abstract: Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves.

By Thiago Sandoval, Ufuk Topcu

llms fine-tuning benchmarks safety

arXiv Machine Learning

Aug 11

From Uncertainty to Failure Attribution: Self-Diagnosing Models for Failure Attribution under Distribution Shift

arXiv:2608. 07953v1 Announce Type: new Abstract: Distribution shift poses a significant challenge to the robustness of machine learning models, but the current solutions only aim to detect out-of-distribution (OOD) samples and predict uncertainty levels.

By Yiyao Yang

benchmarks safety

arXiv Machine Learning

Jun 11

PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

arXiv:2505. 08784v2 Announce Type: replace-cross Abstract: As machine learning (ML) enters high-stakes domains, trustworthy uncertainty quantification (UQ) is essential for safety.

By Abhineet Agarwal, Fange Xiao, Rebecca Barter, Omer Ronen, Boyu Fan, Bin Yu

computer-vision benchmarks safety

One email a morning, machine-written

Related stories

RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

Audited Conformal Prediction for Classification under Unknown Distribution Shift

Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

From Uncertainty to Failure Attribution: Self-Diagnosing Models for Failure Attribution under Distribution Shift

PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework