The paper proposes study contracts that bind experimental choices, run obligations, and claim scope to recorded execution evidence, aiming to distinguish contract-relative verification from scientific truth. A diagnostic with eight clean/mutated pairs shows that a deterministic checker can detect all registered mutations, while a broader test on 18 judge aliases flagged 104 of 144 mutated cases, with remaining cases abstaining or failing. The results highlight an information-asymmetric development setting and suggest further tests—full-information comparisons, legitimate-adaptation controls, and closed-loop agent evaluations—to assess whether contract checks improve compliant completion under optimization.
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
By Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Ruochen Yang, Yingzhi He, Peng Zhang, Jiangxia Cao, Yusheng Huang, Guohong Mu, Jian Liang, Ruiming Tang, Shuang Yang, Zhaojie Liu, Wenwu Ou, Kun Gai
The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks.
"whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
By Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan
arXiv:2606. 17099v1 Announce Type: cross Abstract: AI coding agents increasingly accept assigned software tasks, modify repositories under bounded authority, and return work packages for review.
By Vincent Schmalbach
The paper introduces BenchGuard, a framework that uses advanced language models to audit execution-based agent benchmarks. BenchGuard systematically cross-verifies benchmark artifacts through structured LLM protocols, optionally incorporating agent solutions or execution traces for deeper diagnostics. In tests on ScienceAgentBench and BIXBench Verified-50, BenchGuard uncovered numerous author-confirmed and expert-identified issues, including fatal errors that made tasks unsolvable, and demonstrated cost-effective auditing for complex bioinformatics tasks.
By Xinming Tu (Minta), Tianze Wang (Minta), Yingzhou (Minta), Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi
The paper introduces Runtime Assurance Contracts (RAC) as a formal policy framework for high‑risk AI agents, addressing the "assurance‑transition gap" by binding autonomy boundaries, component eligibility, evidence state, transition policy, human‑review capacity, and non‑compensatory gates. RAC allows soft metrics to influence routing while mandating retries, switches, escalations, deferrals, or stops when mandatory gates fail or are unknown, ensuring aggregate performance cannot alone authorize action. The authors define the contract, evidence record, permission rule, and five invariants, and evaluate RAC through deterministic failure‑injection studies, hand‑authored traces, and a prospective synthetic holdout, comparing it to score‑only and restricted protocol baselines.
By Serhii Zabolotnii
The article argues that agentic auto‑research should be guided by dense, intermediate signals of epistemic progress rather than by sparse final benchmarks. It compares this approach to fuzz testing, where coverage provides continuous feedback that directs input mutation. The authors propose controlled experiments to test whether such signals improve discovery efficiency and reduce false positives, and demonstrate in a simulated physics setting that an AI agent using feedback‑driven search uncovers a hidden law while optimization‑driven baselines fail.
By Yifeng He, Jicheng Wang, Yinzhe Zhao, Chengyang Shi, Jiachen Liu, Hao Chen
arXiv:2606. 02965v2 Announce Type: replace Abstract: As large language models gain tool access and are deployed as autonomous agents capable of editing records, executing transactions, and modifying infrastructure, we still evaluate them based on the sole metric of task completion.
By Victor Ojewale, Suresh Venkatasubramanian
arXiv:2608.31076v1 Announce Type: cross
Abstract: Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experi...
By Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu, Jintian Zhang, Yijun Chen, Zirui Xue, Shumin Deng
The paper discusses the need for independent assurance when coding agents produce both software and its correctness evidence. It identifies four gaps—requirement, conformance, evaluation, and model—between stakeholder intent and production, and proposes a verification framework that separates building from verifying. The framework introduces a Verification Contract, a specification-first phase, and an implementation-aware phase using mutation analysis, property-based testing, and fuzzing, and is evaluated in a study showing that an external verifier detected more seeded faults than the builder’s own tests.
By Sion Israel Sion, Alain April
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
By Hayoung Jung, Pedro Viana Diniz, Jos\'e Reinaldo Corr\^ea Roveda, Abner Fernandes da Silva, Haeun Jung, Enoch Tsai, Aleksandra Korolova, Manoel Horta Ribeiro
Operational reporting for high-performance computing (HPC) facilities requires combining information from monitoring services, schedulers, incident and maintenance records, storage measurements, perio...