arXiv AI
Science sandboxes measure the scientific capability of AI agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 25
The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing
The article argues that agentic auto‑research should be guided by dense, intermediate signals of epistemic progress rather than by sparse final benchmarks. It compares this approach to fuzz testing, where coverage provides continuous feedback that directs input mutation. The authors propose controlled experiments to test whether such signals improve discovery efficiency and reduce false positives, and demonstrate in a simulated physics setting that an AI agent using feedback‑driven search uncovers a hidden law while optimization‑driven baselines fail.
arXiv AI
Oct 2
EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights
EurekaBench is a cross‑domain benchmark designed to assess AI agents’ capacity to discover new scientific insights. It comprises 26 long‑horizon tasks spanning fields such as neuroscience, chemistry, and astrophysics, with 306 expected insights that mechanisms should support. The evaluation measures adherence to scientific constraints, predictive accuracy, and the ability to generate insights or guide future research, revealing that current AI agents excel at prediction but lag in producing meaningful scientific insights.
arXiv AI
Aug 28
Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
The paper "Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research" argues that large language model agents must faithfully implement reference methods, design experiments that truly test claims, and provide supporting evidence. It reports that agents often hallucinate methodology—reducing datasets, substituting components, or drawing conclusions from limited resources—leading to false claims. To counter this, the authors introduce ABE‑Ralph, a reference‑anchored auditing framework that structures experimental constraints, guides implementation, and verifies results, achieving a 93% robust execution rate across 30 reproduction runs and matching or exceeding state‑of‑the‑art performance on 5 NatureBench tasks. "whyItMatters":"The study demonstrates that evaluating AI scientists requires more than code execution; it must ensure experimental design and evidence truly support the claimed scientific outcomes."
arXiv AI
Jun 11
Can AI Agents Synthesize Scientific Conclusions?
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv AI
Sep 16
Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance
arXiv:2609.16213v1 Announce Type: new Abstract: Artificial intelligence is reshaping biological research across an increasingly connected digital-to-physical workflow. General-purpose large language...
arXiv AI
Jun 19
Measuring Biological Capabilities and Risks of AI Agents
arXiv:2606. 19899v1 Announce Type: cross Abstract: This paper addresses a rapidly emerging policy challenge: how to generate and interpret credible evidence about the biological capabilities and risks of AI scientists, or agentic AI systems capable of autonomously or collaboratively performing multi-step scientific tasks.