The paper introduces a framework for online monitoring of AI systems that produce decisions with associated confidence scores. It formulates the problem as an online constrained optimization task, aiming to minimize expert auditing while ensuring that the proportion of unaudited AI errors remains below a user‑defined threshold. The authors present a family of algorithms with provable error‑control guarantees, including an optimally efficient method that achieves the best possible auditing rate for both adversarial and i.i.d. data streams, and demonstrate its effectiveness on tasks such as screening harmful requests, text annotation, and data labeling.
By Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic
The paper introduces a method called continual validation for AI analysts that evolve over time. By evaluating each new version on simulated null data before deployment, it guarantees that the lifetime probability of exceeding a target Type I error rate stays bounded, regardless of how many updates are made. Experiments on bioequivalence studies and A/B tests show that continual validation reduces Type I error rates from 0.098 and 0.183 down to at most 0.045 and 0.044 while preserving most of the statistical power.
By Xiaojing Sun, Yuhan Zeng, Xiao Wang
arXiv:2608. 05490v1 Announce Type: new Abstract: Autonomous agents now carry out entire data analyses, selecting cohorts, joining tables, and fitting models with little step-by-step supervision.
By Ahmed Hassoon, Mark Dredze
arXiv:2610.18948v1 Announce Type: cross
Abstract: Finite safety assessments cannot exclude an adversary that preserves unattacked behavior until certification and activates afterward. We characterize...
By Inderjeet Singh
AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fix...
The paper investigates how AI systems that perform best‑of‑n search require different validation strategies as the search width changes. It shows that auditing only small search widths leaves a gap in reliability estimates for larger widths, and proposes retaining candidate ranks and truth labels to estimate reliability across all widths up to N. The authors derive theoretical bounds on the minimax mean‑squared error, design procedures that achieve these bounds, and demonstrate that a shared audit can significantly reduce maximum error across many widths in practical CodeRM pools.
By Ricardo Fitas