BioStudyBench is a new benchmark that tests AI agents on 25 long‑horizon analysis tasks derived from biomedical studies published between July and September 2026, after the models’ knowledge cutoffs. Each task gives the agent a neutral research question but no data files, requiring it to locate and download relevant public data, search literature using tools limited to pre‑cutoff records, and perform data analysis to report findings. The benchmark shows that providing data and tools improves model performance by an average of 47 percentage points, with closed‑weight models outperforming open‑weight ones (best open‑weight 81.3% vs. best closed‑weight 94.7%).
We evaluate whether AI agents can match the reported findings of published biomedical studies using public data. Existing evaluations do not consistently separate analysis from prior knowledge or retr...
arXiv:2605. 06177v2 Announce Type: replace Abstract: Reproducing and comparing deep research agents today is hard: the same backbone evaluated on the same benchmark can report different accuracies across papers because the harness and tool registry differ, and integrating a new model into a comparable evaluation surface costs weeks of model-specific engineering.
arXiv:2610.29812v1 Announce Type: cross Abstract: Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled...
arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.
arXiv:2606. 29746v1 Announce Type: new Abstract: Navigating the deluge of heterogeneous medical data, from academic literature (PubMed) to clinical guidelines (Web) and private knowledge bases, remains a critical bottleneck for evidence-based medicine.
The paper introduces SEER‑Repro, a benchmark of 20 SEER‑based studies with data and resources for reproduction. An LLM agent reproduces 77.4% of 2,190 atomic claims and provides executable baselines. An agentic analysis pipeline then tests robustness by identifying analytical choices, proposing alternatives, and rerunning analyses, flagging potentially consequential choices in 9.5% of fully reproduced findings for expert review.
arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.
arXiv:2604. 16742v2 Announce Type: replace Abstract: Scientists have long sought to accurately predict outcomes of real-world events before they happen.
TruthInsightBench is a new benchmark designed to evaluate automated scientific discovery agents by presenting them with 40 blind tasks drawn from peer‑reviewed studies across ten domains. Each task provides only a neutral objective and frozen data, withholding source conclusions, expected values, and analysis paths, forcing agents to determine which claim the data support. A fixed LLM‑based judge scores agents on evidentiary maturity across six dimensions, using 29 artifact‑grounded items, enabling fully automated, repeatable evaluation without human grading.
Deep Research Bench II is a new benchmark designed to evaluate Deep Research Agents (DRAs) by requiring them to produce research reports for 132 grounded tasks across 22 domains. Each report is assessed using 9,430 fine‑grained binary rubrics that cover information recall, analysis, and presentation, all derived from expert‑written investigative articles through a rigorous LLM‑plus‑human pipeline. Evaluation of current state‑of‑the‑art DRAs shows that even the best models satisfy fewer than 50% of these rubrics, highlighting a significant gap between automated agents and human experts.
ReviewPilot is a conversational multi‑agent system designed to automate the PRISMA workflow for systematic literature reviews, covering both paper screening and evidence extraction. Researchers can customize screening criteria and extraction schemas through dialogue, with a project memory preserving configurations and revisions. The system records the rationale for each screening decision and the source of every extracted value, and it has demonstrated superior performance to single‑prompt LLMs and AgentSLR in epidemiology, achieving screening macro F1 scores up to 0.776 and extraction F1 up to 0.788, while also showing close agreement with human annotators in biomedicine and human‑computer interaction case studies.
Navigating the deluge of heterogeneous medical data, from academic literature (PubMed) to clinical guidelines (Web) and private knowledge bases, remains a critical bottleneck for evidence-based medicine. While commercial black-box tools lack transparency, standard open-source RAG implementations frequently suffer from reasoning drift when handling complex, long-tail queries.