arXiv Machine Learning
Jul 10
Prediction-Powered Active Testing
arXiv:2607. 08347v1 Announce Type: cross Abstract: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled.
Prediction-Powered Active Testing
Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive.
Related stories
arXiv Machine Learning
Jun 11
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
arXiv:2602. 02229v2 Announce Type: replace Abstract: We study the problem of monitoring model performance in dynamic environments where labeled data are limited.
arXiv Machine Learning
Jun 26
No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference
arXiv:2505. 20178v2 Announce Type: replace-cross Abstract: Prediction-Powered Inference (PPI) is a popular strategy for combining gold-standard and possibly noisy pseudo-labels to perform statistical estimation.
arXiv Machine Learning
Jun 15
Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
arXiv:2606. 14506v1 Announce Type: cross Abstract: Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making.
arXiv Machine Learning
Jun 26
Learning from a Biased Sample
arXiv:2209. 01754v5 Announce Type: replace-cross Abstract: The empirical risk minimization approach to data-driven decision making requires access to training data drawn under the same conditions as those that will be faced when the decision rule is deployed.
arXiv AI
Jun 3
ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
arXiv:2604. 23099v2 Announce Type: replace-cross Abstract: Evaluating generative AI models is increasingly resource-intensive due to slow inference, expensive raters, and a rapidly growing landscape of models and benchmarks.
arXiv Machine Learning
Aug 11
Demystifying Prediction Powered Inference
arXiv:2601. 20819v2 Announce Type: replace-cross Abstract: Machine learning predictions are increasingly used to supplement incomplete or costly-to-measure outcomes in fields such as biomedical research, environmental science, and social science.
arXiv Machine Learning
Jun 16
Audited Conformal Prediction for Classification under Unknown Distribution Shift
arXiv:2606. 14909v1 Announce Type: cross Abstract: We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift.
arXiv Machine Learning
Jun 30
Conformal Prediction with Macro-Coverage Guarantees
arXiv:2606. 28598v1 Announce Type: cross Abstract: Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others.
arXiv Machine Learning
1d ago
Inferential Evaluation of Surrogate-Derived Models under Covariate Shift
arXiv:2608. 15783v1 Announce Type: cross Abstract: In transfer-learning settings, a model derived from abundant surrogate labels may be deployed in a target population where gold-standard outcomes are unobserved.
arXiv Machine Learning
Jun 25
Efficient Analytic Uncertainty Quantification for Multi-Modal Regression
arXiv:2606. 25188v1 Announce Type: new Abstract: Efficient uncertainty quantification (UQ) is essential for trustworthy large-scale learning.
arXiv Machine Learning
Jul 27
Smart predict-then-robustly-optimize
arXiv:2607. 21773v1 Announce Type: new Abstract: In this paper, we propose and study a robust variant of the smart predict-then-optimize approach that accounts for prediction shifts due to disturbance in the covariate feature space.