The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks.
"whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
By Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan
arXiv:2606. 31478v1 Announce Type: new Abstract: Autonomous research agents can now draft hypotheses, write code, run experiments, and produce papers, but they remain brittle when experiments fail.
By Jie Ma, Binfei Chu, Jie Gao, Jinlu Zhang, Yiwei Ma, Yi Tan, Jiayi Ji, Xiaoshuai Sun, Rongrong Ji
arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.
By Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Mingkai Zhang, Yanming Zhu
VeriHarness is a method that enhances verification for large language model agents tackling long‑horizon tasks without needing reference answers at test time. It transforms the base LLM into an agentic verifier by providing a workspace, evidence tools, and reusable verification skills, using disagreement resolution and consensus challenge to evaluate competing claims. Across five benchmarks and two frontier models, VeriHarness outperforms baselines, achieving significant performance gains and demonstrating self‑improvement of verification skills from failure feedback.
By Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, Chen-Yu Lee
arXiv:2607. 02134v1 Announce Type: new Abstract: Scientific machine learning papers typically make computational claims, e.
By Atharva Hans, Ilias Bilionis
arXiv:2608. 03674v1 Announce Type: new Abstract: Causal diagnostic models must explain how conclusions follow from evidence because diagnoses guide repairs and treatments.
By Jian Zhang, Bingyi Wang, Yizhi Liu
The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.
By Peiying Zhu, Sidi Chang
Causal diagnostic models must explain how conclusions follow from evidence because diagnoses guide repairs and treatments. Yet serious cases are scarce, records rarely contain reasoning paths, and data transfer poorly across configurations, complicating local deployment.
The paper introduces ToxicBench, a benchmark designed to evaluate how tool‑augmented data agents handle incorrect tool outputs. By pairing clean and poisoned observations across numerical, label, schema, and retrieval errors, the authors assess both the checking process and the final answer adoption. In a 118‑task GPT evaluation, poisoning reduces task success by 26–39 percentage points, revealing that repeated poisoning leads to wrong-answer adoption even after checking, while ordinary retries help only under one‑shot poisoning. Human annotations on 200 trajectories confirm the scoring system’s reliability, showing 96% agreement with task success and supporting the benefits of retries and audit‑based adoption.
By Zifu Tao, Changqing Yin
arXiv:2609.24264v1 Announce Type: new
Abstract: Tool-use agent traces identify messages and API calls, but procedural analyses also need explicit units of action and inspectable links to their eviden...
By Songqi Li, Dongqing Li, Zheqiao Cheng
arXiv:2607. 20827v1 Announce Type: new Abstract: LLM agents choose tools and arguments from context that mixes user requests, tool outputs, retrieved records, memory, and untrusted text.
By Junchi Liao
arXiv:2607. 09682v1 Announce Type: new Abstract: AI systems are increasingly used to assist consequential decisions in regulated domains such as auditing, finance, and healthcare.
By Vimal Nakrani