A single human must audit $N$ LLM agents under a budget of $B \ll N$ audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $δ^*$ past which confidence-ranked auditing is \emph{worse} than random.
arXiv:2610.09371v1 Announce Type: cross Abstract: Calibrated uncertainty quantification is essential to ensuring AI agents are trustworthy and reliable. However, when agents seek to maximize user eng...
arXiv:2610.18780v1 Announce Type: new Abstract: Large language model multi-agent systems extend individual agents' capabilities through collaboration across specialized roles, tools, and workflows. Y...
The paper introduces DISCERN, a two-tier protocol for certifying that updates to production models do not increase risk. It first uses unlabeled data to detect benign updates based on disagreement rates, then selectively labels only disagreements through an anytime-valid confidence sequence. The method achieves finite-sample validity with label-complexity bounds of order ρ²/ε², demonstrating significant label savings and strong empirical performance across 14,000+ audit streams.
arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...
arXiv:2609.15122v1 Announce Type: cross Abstract: Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as...
The paper introduces a hypothesis testing framework for adaptive auditing of AI systems, addressing the challenge of limited, opportunistically collected data. It defines two competing null hypotheses—one from the model’s perspective and one from the auditor’s—and uses Safe Anytime-Valid Inference (SAVI) to perform betting-based tests that yield simultaneous e-processes. The authors prove that, with a powerful auditor, the two hypotheses become asymptotically inverse, allowing a global robustness certification, and demonstrate empirically that their methods control Type‑I error and can reach conclusions with as few as 20 observations.
The paper investigates whether improved confidence estimates can enhance the audit of knowledge graphs constructed by large language models (LLMs). It models non‑adaptive auditing as a budgeted covering problem, identifies key error statistics, and bounds the regret of scalar‑preserving planning. Experiments on Re‑DocRED and cross‑document graphs show that while better error prediction offers limited gains, joint planning and LLM verification can reduce residual errors by up to about 1.5 percentage points.
arXiv:2610.33112v1 Announce Type: new Abstract: Use-based AI regulation assigns obligations by deployment risk but lacks an evidentiary object to carry them. A benchmark score is a property of a mode...
arXiv:2607. 19692v1 Announce Type: cross Abstract: Observational causal analyses increasingly pool records across sites, vendors, and collection systems, creating vulnerability to append-only attacks in which plausible records are strategically selected to alter a reported treatment effect.
The paper introduces a framework for online monitoring of AI systems that produce decisions with associated confidence scores. It formulates the problem as an online constrained optimization task, aiming to minimize expert auditing while ensuring that the proportion of unaudited AI errors remains below a user‑defined threshold. The authors present a family of algorithms with provable error‑control guarantees, including an optimally efficient method that achieves the best possible auditing rate for both adversarial and i.i.d. data streams, and demonstrate its effectiveness on tasks such as screening harmful requests, text annotation, and data labeling.
arXiv:2603. 13356v2 Announce Type: replace Abstract: Robust reinforcement learning typically assumes that feedback sources are either globally trustworthy or corrupted within a fixed global budget.
arXiv:2608. 05490v1 Announce Type: new Abstract: Autonomous agents now carry out entire data analyses, selecting cohorts, joining tables, and fitting models with little step-by-step supervision.