The paper introduces a hypothesis testing framework for adaptive auditing of AI systems, addressing the challenge of limited, opportunistically collected data. It defines two competing null hypotheses—one from the model’s perspective and one from the auditor’s—and uses Safe Anytime-Valid Inference (SAVI) to perform betting-based tests that yield simultaneous e-processes. The authors prove that, with a powerful auditor, the two hypotheses become asymptotically inverse, allowing a global robustness certification, and demonstrate empirically that their methods control Type‑I error and can reach conclusions with as few as 20 observations.
By Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman, Yifan Mai, Jean Feng
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.
By Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth
The paper introduces a framework for online monitoring of AI systems that produce decisions with associated confidence scores. It formulates the problem as an online constrained optimization task, aiming to minimize expert auditing while ensuring that the proportion of unaudited AI errors remains below a user‑defined threshold. The authors present a family of algorithms with provable error‑control guarantees, including an optimally efficient method that achieves the best possible auditing rate for both adversarial and i.i.d. data streams, and demonstrate its effectiveness on tasks such as screening harmful requests, text annotation, and data labeling.
By Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic
The paper introduces a policy‑conditioned AI‑use detection framework for academic publishing, arguing that traditional AI detection tools misalign with venue rules by merely identifying AI‑generated text. Instead, the proposed system treats the governing policy as an explicit input, generating hypotheses, evidence, calibration, and uncertainty rather than binary verdicts. It outlines how to benchmark compliance, evaluate true positive rates at venue‑specified false positive thresholds, and emphasizes the need for structured disclosure, tool routing, and contestable findings.
By Jairo Diaz-Rodriguez, Mumin Jia
The paper introduces a framework for hypothesis testing that combines inexpensive AI judgments with selective human verification to control type‑I and type‑II errors while minimizing cost. It derives an information‑theoretic lower bound on the minimum cost and proposes the SCALE policy, a sequential, cost‑aware strategy that adapts AI scoring and human escalation. SCALE is proven valid for finite samples and asymptotically matches the lower bound, achieving significant savings when both AI and human inputs are valuable.
By Dae Woong (David), Ham, Xuejun Zhao, Stefanus Jasin, Fenghua Yang
arXiv:2605.29139v2 Announce Type: replace-cross
Abstract: Question-answering services built on retrieval-augmented generation (RAG), in which a language model answers from retrieved documents, are in...
By Prasanjit Dubey, Xiaoming Huo