arXiv:2608.30165v1 Announce Type: cross
Abstract: Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to desig...
By Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai, Kenneth B. Hsu, Yasha Ektefaie, Shantanu Singh, Sangeeta N. Bhatia, Steven K. Reilly, Ryan Tewhey, Eric S. Lander, Pardis C. Sabeti
arXiv:2609.39140v1 Announce Type: new
Abstract: Learning to complete tasks in unfamiliar environments with unknown rules remains a key challenge for LLM agents. Current LLM agents often record their...
By Guanning Zeng, Jiani Wang, Wenjie Ma, Shaofeng Yin, Chenyang Wang, Shichen Liu, Angjoo Kanazawa, Wode Ni, Xiuyu Li, Andrea Zanette, Haiwen Feng
arXiv:2607. 04293v1 Announce Type: cross Abstract: Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention.
By Zhenhao Chen, Yongqiang Chen, Chenxi Liu, Junchi Yu, Xiangchen Song, Zijian Li, Jialin Li, Philip Torr, Bo Han, Kun Zhang
The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks.
"whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
By Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan
The paper introduces LLM-IDEA, an identifiability‑driven experimental agent that guides large language model scientists in closed‑loop discovery of mechanistic world models. It incorporates an identifiability engine that classifies experimental plateaus as capability limits, resolvable within the design class, or certified exhausted, and demonstrates its effectiveness on benchmarks such as ODEBench, DiscoverPhysics, and a custom Alien Universe testbed. The agent’s ability to distinguish when further experimentation is futile versus when new designs are needed improves the efficiency and reliability of autonomous scientific discovery.
By Surya Shetty, Ulisses Braga-Neto
SciGame is a new benchmark that transforms scientific literature into structured cards, each providing both an Original and a Counterfactual environment for the same task. It evaluates language‑model agents in identification and discovery scenarios under Prior‑only, Fixed, and Adaptive protocols, using historical canonical laws and a live temporal collection. Experiments with five advanced models reveal that prior knowledge can hurt performance when relations change, adaptive experimentation benefits strong models but not weak ones, and accurate explanations may mask quantitative errors, highlighting the difficulty of recovering exact symbolic laws.
By Yusheng Zhao, Bowei He, Xue Liu, Mingxiao Li