The paper introduces a method called continual validation for AI analysts that evolve over time. By evaluating each new version on simulated null data before deployment, it guarantees that the lifetime probability of exceeding a target Type I error rate stays bounded, regardless of how many updates are made. Experiments on bioequivalence studies and A/B tests show that continual validation reduces Type I error rates from 0.098 and 0.183 down to at most 0.045 and 0.044 while preserving most of the statistical power.
AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fix...
The paper introduces a hypothesis testing framework for adaptive auditing of AI systems, addressing the challenge of limited, opportunistically collected data. It defines two competing null hypotheses—one from the model’s perspective and one from the auditor’s—and uses Safe Anytime-Valid Inference (SAVI) to perform betting-based tests that yield simultaneous e-processes. The authors prove that, with a powerful auditor, the two hypotheses become asymptotically inverse, allowing a global robustness certification, and demonstrate empirically that their methods control Type‑I error and can reach conclusions with as few as 20 observations.
The article argues that agentic auto‑research should be guided by dense, intermediate signals of epistemic progress rather than by sparse final benchmarks. It compares this approach to fuzz testing, where coverage provides continuous feedback that directs input mutation. The authors propose controlled experiments to test whether such signals improve discovery efficiency and reduce false positives, and demonstrate in a simulated physics setting that an AI agent using feedback‑driven search uncovers a hidden law while optimization‑driven baselines fail.
arXiv:2606. 08106v1 Announce Type: new Abstract: Self-evolving agents improve by repeatedly proposing changes to their own prompts, skills, or workflows and keeping those that score higher on a small held-out set.
arXiv:2609.33180v2 Announce Type: replace-cross Abstract: As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies...
arXiv:2608. 02636v1 Announce Type: cross Abstract: Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model.
The paper introduces VERSE, a Verified Self‑Evolving optimizer that enhances LLM agent harnesses by allowing the optimizer to test edits, replay failures, and perturb steps while tracking fixes and regressions. VERSE builds its own tools for failure analysis, verification, training audits, and workflow control, and uses this feedback to revise the harness’s prompts, skills, tools, hooks, and notes without changing model weights. In experiments across five executors and multiple languages, VERSE improves all evaluated harness optimizers, achieving higher accuracy on held‑out and out‑of‑distribution tasks compared to the strongest baselines.
The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
arXiv:2608. 09855v1 Announce Type: new Abstract: Autonomous research agents can generate experiments faster than researchers can validate them.
arXiv:2607. 28666v1 Announce Type: cross Abstract: Enterprise AI programmes stall at a rate that is widely quoted and poorly explained.
arXiv:2609.16302v1 Announce Type: cross Abstract: When a coding agent returns to existing software, it inherits evidence from earlier engineering work: tests, type checks, proofs, static analyses, an...
The paper investigates the reliability of software produced by agentic AI by comparing AI-generated versions of ten well-known Linux utilities to their human-written counterparts. Using fuzz testing (both black-box and coverage-guided AFL++), the authors find that AI-generated code is often as reliable or more reliable than the latest human versions, with fewer memory errors but a higher incidence of hangs. The study emphasizes that robust AI-generated software requires careful prompting, skilled human oversight, and that the AI workflow can serve as a cost-effective specification for sustainable code.