The paper introduces the Science Prediction Universe (SPU), a benchmark comprising 100 public verified tasks and 16 private wet‑lab tasks across chemistry, materials science, biology, and medicine, designed to test agents’ ability to predict experimental outcomes. Thirteen agent configurations using twelve frontier models were evaluated, revealing significant performance gaps: Spearman correlations ranged from 0.227 to 0.478 on the public set. Analysis of over a thousand trajectories indicates that current agents struggle more with scientific judgment than tool execution, often relying on shared priors and trivial proxies rather than accurate predictions.
arXiv:2610.18449v1 Announce Type: new Abstract: AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide f...
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
The paper introduces Autonomous Science Bench, a benchmark for AI agents that must choose experiments under a limited budget. It features six real-world design-build-test-learn campaigns in biology, chemistry, and materials science, where agents select design batches, receive lab measurements, and iteratively refine a predictive model. Success is measured by two calibrated gates—discovery and learning—highlighting distinct agent weaknesses that overall success rates alone would mask.
arXiv:2609.33357v2 Announce Type: replace Abstract: Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially red...
EurekaBench is a cross‑domain benchmark designed to assess AI agents’ capacity to discover new scientific insights. It comprises 26 long‑horizon tasks spanning fields such as neuroscience, chemistry, and astrophysics, with 306 expected insights that mechanisms should support. The evaluation measures adherence to scientific constraints, predictive accuracy, and the ability to generate insights or guide future research, revealing that current AI agents excel at prediction but lag in producing meaningful scientific insights.
arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...
The paper introduces SciConBench, a large-scale benchmark with 9.21K health-related questions and expert-written conclusions, designed to evaluate AI agents’ ability to synthesize scientific conclusions. It employs an automated evaluation pipeline that breaks conclusions into atomic facts and measures factual precision and recall, and introduces SciConHarness, a clean‑room harness that limits web interaction to prevent data leakage. Experiments on eight frontier models and deep research agents show low factual quality (best F1 = 0.337) and that clean‑room settings consistently reduce performance, indicating that leakage inflates perceived capabilities. The study also audits consumer‑facing agents, finding they often produce incomplete or contradictory conclusions even when the correct answer is available.
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
OpenDiscoveryTrace is a public dataset of 558 complete AI scientific agent trajectories that records the reasoning process—thoughts, tool calls, observations, errors, revision triggers, and confidence—across 124 scientific tasks in drug discovery, materials science, genomics, and literature analysis. The dataset includes seven models (three frontier models and four open‑weight models) and 60 live‑retrieval variants, providing a balanced view of performance and error patterns. Pilot analysis shows that process traces reveal behavioral differences invisible to output‑only evaluation, such as differing error rates and types among frontier models.
arXiv:2608.30165v1 Announce Type: cross Abstract: Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to desig...
The paper introduces ScientistTwo, a fully autonomous multi‑agent framework that takes a scientific problem, establishes baselines, generates hypotheses, and coordinates specialized agents to conduct an end‑to‑end discovery cycle without human intervention. It rigorously tests and refines its methods through automated experiments, ablation studies, and a closed‑loop peer‑review engine. Benchmarking against top conferences (ICLR, ICML, NeurIPS) shows that ScientistTwo produces expert‑level, publishable papers and codebases that outperform human state‑of‑the‑art models and receive higher review ratings under automated AI review.