The paper introduces a framework for online monitoring of AI systems that produce decisions with associated confidence scores. It formulates the problem as an online constrained optimization task, aiming to minimize expert auditing while ensuring that the proportion of unaudited AI errors remains below a user‑defined threshold. The authors present a family of algorithms with provable error‑control guarantees, including an optimally efficient method that achieves the best possible auditing rate for both adversarial and i.i.d. data streams, and demonstrate its effectiveness on tasks such as screening harmful requests, text annotation, and data labeling.
By Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic
The paper introduces a method called continual validation for AI analysts that evolve over time. By evaluating each new version on simulated null data before deployment, it guarantees that the lifetime probability of exceeding a target Type I error rate stays bounded, regardless of how many updates are made. Experiments on bioequivalence studies and A/B tests show that continual validation reduces Type I error rates from 0.098 and 0.183 down to at most 0.045 and 0.044 while preserving most of the statistical power.
By Xiaojing Sun, Yuhan Zeng, Xiao Wang
arXiv:2608. 05490v1 Announce Type: new Abstract: Autonomous agents now carry out entire data analyses, selecting cohorts, joining tables, and fitting models with little step-by-step supervision.
By Ahmed Hassoon, Mark Dredze
arXiv:2610.18948v1 Announce Type: cross
Abstract: Finite safety assessments cannot exclude an adversary that preserves unattacked behavior until certification and activates afterward. We characterize...
By Inderjeet Singh
AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fix...
The paper investigates how AI systems that perform best‑of‑n search require different validation strategies as the search width changes. It shows that auditing only small search widths leaves a gap in reliability estimates for larger widths, and proposes retaining candidate ranks and truth labels to estimate reliability across all widths up to N. The authors derive theoretical bounds on the minimax mean‑squared error, design procedures that achieve these bounds, and demonstrate that a shared audit can significantly reduce maximum error across many widths in practical CodeRM pools.
By Ricardo Fitas
The paper presents two finite-sample conformal methods for controlling false alerts in AI-generated text screening. Construction A assigns a false-alert budget across a finite set of prefix‑detector scores, using a union bound to protect any executed subset. Construction B calibrates the maximum of a development‑fixed adaptive policy, ensuring that early stopping is protected by a terminal conformal rank without splitting the error budget. Both approaches guarantee marginal false‑alert control under document‑level exchangeability and provide calibration requirements for rejection, while noting that detection power and computational efficiency require empirical study.
By Marco Mandap, Jerahmeel Hipolito, Arcel Galvez, Charlie Margaret Balagtas, Michael Joshua Buluran, Jeff Roel Durmiendo, Rizzette E. Lopez
The paper proposes a claim‑specific verification audit for modular agents that replaces aggregate task scores with evidence‑based evaluations. Each agent conclusion is recorded with supporting evidence and classified as supported, unsupported, unresolved, or not evaluated, along with the boundary of validity. The audit employs three tools—oracle policies, perfect component replacements, and verifier‑score tests—to trace value changes, locate lost value, and assess verifier effectiveness, demonstrated on a portfolio‑allocation agent in a synthetic market.
By Ali Atiah Alzahrani
arXiv:2608.22160v1 Announce Type: new
Abstract: Physical automation is scaling toward fleets of embodied machines commanded by an AI brain. Early deployments already run factories and warehouses at p...
By Zhixu Du, Yiran Chen
The paper introduces a framework for hypothesis testing that combines inexpensive AI judgments with selective human verification to control type‑I and type‑II errors while minimizing cost. It derives an information‑theoretic lower bound on the minimum cost and proposes the SCALE policy, a sequential, cost‑aware strategy that adapts AI scoring and human escalation. SCALE is proven valid for finite samples and asymptotically matches the lower bound, achieving significant savings when both AI and human inputs are valuable.
By Dae Woong (David), Ham, Xuejun Zhao, Stefanus Jasin, Fenghua Yang
arXiv:2606. 30338v1 Announce Type: new Abstract: External evaluations are becoming increasingly central to the governance of AI systems.
By Ioannis Pitsiorlas, Martha V. Sourla, Marios Kountouris
TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.
By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song