The paper introduces SciConBench, a large-scale benchmark with 9.21K health-related questions and expert-written conclusions, designed to evaluate AI agents’ ability to synthesize scientific conclusions. It employs an automated evaluation pipeline that breaks conclusions into atomic facts and measures factual precision and recall, and introduces SciConHarness, a clean‑room harness that limits web interaction to prevent data leakage. Experiments on eight frontier models and deep research agents show low factual quality (best F1 = 0.337) and that clean‑room settings consistently reduce performance, indicating that leakage inflates perceived capabilities. The study also audits consumer‑facing agents, finding they often produce incomplete or contradictory conclusions even when the correct answer is available.
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
PAPERBENCHX is a new benchmark that tests whether AI can reproduce scientific experiments across 12 disciplines, including electromagnetics, photonics, chemistry, materials science, biology, and robotics. It consists of 93 expert-constructed tasks, each providing a source paper, a public specification, and a domain-native scientific environment. The benchmark evaluates AI agents by having them reconstruct, execute, and diagnose key experiments, then replay submissions from scratch to score regenerated evidence against expert-validated criteria, revealing that current AI configurations fully reproduce only 14% of tasks.
The paper introduces SEER‑Repro, a benchmark of 20 SEER‑based studies with data and resources for reproduction. An LLM agent reproduces 77.4% of 2,190 atomic claims and provides executable baselines. An agentic analysis pipeline then tests robustness by identifying analytical choices, proposing alternatives, and rerunning analyses, flagging potentially consequential choices in 9.5% of fully reproduced findings for expert review.
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
arXiv:2609.33357v2 Announce Type: replace Abstract: Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially red...
The paper introduces the Science Prediction Universe (SPU), a benchmark comprising 100 public verified tasks and 16 private wet‑lab tasks across chemistry, materials science, biology, and medicine, designed to test agents’ ability to predict experimental outcomes. Thirteen agent configurations using twelve frontier models were evaluated, revealing significant performance gaps: Spearman correlations ranged from 0.227 to 0.478 on the public set. Analysis of over a thousand trajectories indicates that current agents struggle more with scientific judgment than tool execution, often relying on shared priors and trivial proxies rather than accurate predictions.
arXiv:2606. 12736v1 Announce Type: new Abstract: AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood.
arXiv:2607. 05456v1 Announce Type: new Abstract: While recent advances in large language models have enabled end-to-end automated manuscript generation, existing systems suffer from three critical deficiencies: (i) generated claims are not deterministically grounded in verifiable literature, (ii) experimental results are frequently fabricated rather than executed, and (iii) there exists no standardized, multi-dimensional framework to assess whether AI-generated manuscripts meet the quality and rigor required for real-world publication.
We evaluate whether AI agents can match the reported findings of published biomedical studies using public data. Existing evaluations do not consistently separate analysis from prior knowledge or retr...
NIBench is a neuroimaging benchmark that evaluates autonomous AI agents on three fronts: execution and workflow reconstruction across 56 tasks in seven imaging modalities; skill evolution through distillation of execution trajectories; and scientific judgment via analysis of 36 Alzheimer’s disease claims. The study tests 19 language‑model agents, finding that top performers execute protocols nearly flawlessly, while distilled skills boost performance on known tasks but yield mixed results on unseen ones. Agents can reproduce reported results yet sometimes draw conclusions that diverge from expert answers, indicating that data access alone does not guarantee accurate scientific judgment.
BioStudyBench is a new benchmark that tests AI agents on 25 long‑horizon analysis tasks derived from biomedical studies published between July and September 2026, after the models’ knowledge cutoffs. Each task gives the agent a neutral research question but no data files, requiring it to locate and download relevant public data, search literature using tools limited to pre‑cutoff records, and perform data analysis to report findings. The benchmark shows that providing data and tools improves model performance by an average of 47 percentage points, with closed‑weight models outperforming open‑weight ones (best open‑weight 81.3% vs. best closed‑weight 94.7%).
arXiv:2409. 11363v2 Announce Type: replace-cross Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research.