arXiv Machine Learning
Jul 10
Prediction-Powered Active Testing
arXiv:2607. 08347v1 Announce Type: cross Abstract: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled.
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
arXiv:2602. 02229v2 Announce Type: replace Abstract: We study the problem of monitoring model performance in dynamic environments where labeled data are limited.
Related stories
Hugging Face Trending Papers
Jul 9
Prediction-Powered Active Testing
Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive.
arXiv AI
Aug 17
Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers
arXiv:2608. 14089v1 Announce Type: new Abstract: Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves.
arXiv Machine Learning
Jun 15
Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias
arXiv:2606. 14506v1 Announce Type: cross Abstract: Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making.
arXiv Machine Learning
Jun 3
BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks
arXiv:2606. 02947v1 Announce Type: new Abstract: Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks.
arXiv AI
Jun 11
Estimating Tail Risks in Language Model Output Distributions
arXiv:2604. 22167v2 Announce Type: replace-cross Abstract: Language models are increasingly capable and are being rapidly deployed on a population-level scale.
arXiv Machine Learning
Jun 16
Audited Conformal Prediction for Classification under Unknown Distribution Shift
arXiv:2606. 14909v1 Announce Type: cross Abstract: We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift.
arXiv Machine Learning
Jul 30
Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
arXiv:2607. 26820v1 Announce Type: new Abstract: As large language models (LLMs) evolve from standalone assistants into autonomous agents, ensuring their safety requires shifting beyond pointwise risk assessment to understand how risks emerge and unfold over long-horizon trajectories.
arXiv Machine Learning
Jul 14
Monitoring the calibration of probability forecasts with an application to concept drift detection involving image classification
arXiv:2510. 25573v2 Announce Type: replace-cross Abstract: Machine learning approaches for image classification have led to impressive advances in that field.
arXiv Machine Learning
Sep 14
PLSP (Pre-hoc Liminal Space Profiling): OOD Prediction over Detection -- An Anticipatory Approach for Machine Learning Model Reliability
The paper introduces PLSP (Pre-hoc Liminal Space Profiling), an anticipatory framework for predicting out-of-distribution (OOD) data before inference. It proposes a dataset‑independent metric called the CREDibility Score (CREDS) and introduces credibility curves and heat maps to analyze a model’s maximum credibility and behavior across datasets. Experiments on multiple datasets show that CREDS can improve model robustness to OOD prediction.
arXiv Statistics ML
Sep 23
Context-Adaptive Thresholding for Conditionally Representative Monitoring and Classification
arXiv:2609.26652v1 Announce Type: cross Abstract: Commonly, classifiers and monitoring procedures are trained from labeled data by optimizing an objective such as the misclassification rate. This may...
arXiv AI
Jul 8
Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets
arXiv:2607. 05516v1 Announce Type: cross Abstract: Model-specific adversarial attacks have been extensively studied.