ReviewPilot is a conversational multi‑agent system designed to automate the PRISMA workflow for systematic literature reviews, covering both paper screening and evidence extraction. Researchers can customize screening criteria and extraction schemas through dialogue, with a project memory preserving configurations and revisions. The system records the rationale for each screening decision and the source of every extracted value, and it has demonstrated superior performance to single‑prompt LLMs and AgentSLR in epidemiology, achieving screening macro F1 scores up to 0.776 and extraction F1 up to 0.788, while also showing close agreement with human annotators in biomedicine and human‑computer interaction case studies.
arXiv:2603. 22327v2 Announce Type: replace-cross Abstract: Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs).
SciLitBench is a multi-stage benchmark for evaluating large language models (LLMs) in systematic literature reviews, covering title and abstract screening, full-text screening, and schema-guided data extraction across 42,981 records and 888 included papers. The study shows that explicit inclusion/exclusion criteria boost title and abstract screening performance by 28.8% and researcher-authored rationales improve full-text screening by 15%. Data extraction performance varies widely, with high accuracy for publication year but low overlap for computational approaches, and even the best models recover only a fraction of annotated evidence and limitations.
arXiv:2606. 28363v1 Announce Type: cross Abstract: Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points.
The paper presents a pipeline that uses Large Language Models (LLMs) to extract information from 536 peer‑reviewed agent‑based modeling papers for systematic literature reviews (SLRs). GPT‑4.1 achieves about 77.95% paper‑level accuracy, while GPT‑5.0 reaches 81.67%. Field‑level accuracy varies widely, and the study notes that agreement between LLMs can signal output quality, with low agreement indicating hallucinations and high agreement with low accuracy suggesting noise in the human dataset.
EviStreams is a live, open‑source, no‑code web platform that enables systematic review teams to control AI‑assisted data extraction at three stages: program design, field specification, and extracted predictions. Reviewers use a form builder to define typed fields, run extraction on PDFs, inspect AI‑generated values with supporting passages, and perform blinded dual review with adjudication to produce an auditable consensus export. An evaluation across four clinical corpora and three model families shows that extraction quality depends more on field specification than on the model choice.
arXiv:2606. 28362v1 Announce Type: cross Abstract: Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort.
arXiv:2606. 19345v1 Announce Type: cross Abstract: The rapid increase in scientific publications leads to the fact that manual study screening in systematic literature reviews (SLRs) is increasingly resource consuming, inefficient, and inconsistent.
The paper introduces SRBench, a benchmark dataset comprising 45,064 labeled entries from 32 curated secondary studies, designed to evaluate large language model performance in systematic review screening while addressing class imbalance. It also presents PromptSR, a tool that facilitates prompt experimentation, experiment management, and result analysis for LLM-based screening. A use case demonstrates the practical application of both SRBench and PromptSR.
The paper introduces BioCheck Agent, an LLM-based system that generates structured biomedical fact‑checking reports using agentic search and a reinforcement‑learning framework called EG‑GRPO. Unlike prior methods that output only supported or refuted labels, BioCheck Agent synthesizes conclusions with retrieved evidence from PubMed, employing advanced Boolean search operators. Experiments show that, compared to the base Qwen3.5‑4B model, BioCheck Agent improves label prediction accuracy on SciFact by 9.95 %, raises evidence quality by 3.7 %, and reduces hallucinations by 19.63 %.
BioStudyBench is a new benchmark that tests AI agents on 25 long‑horizon analysis tasks derived from biomedical studies published between July and September 2026, after the models’ knowledge cutoffs. Each task gives the agent a neutral research question but no data files, requiring it to locate and download relevant public data, search literature using tools limited to pre‑cutoff records, and perform data analysis to report findings. The benchmark shows that providing data and tools improves model performance by an average of 47 percentage points, with closed‑weight models outperforming open‑weight ones (best open‑weight 81.3% vs. best closed‑weight 94.7%).
The paper introduces SEER‑Repro, a benchmark of 20 SEER‑based studies with data and resources for reproduction. An LLM agent reproduces 77.4% of 2,190 atomic claims and provides executable baselines. An agentic analysis pipeline then tests robustness by identifying analytical choices, proposing alternatives, and rerunning analyses, flagging potentially consequential choices in 9.5% of fully reproduced findings for expert review.
ClinAgent is a conversational system that uses a ReAct-based LLM agent to retrieve and synthesize clinical trial information from multiple sources such as ClinicalTrials.gov, PubMed, and a local dataset. The agent iteratively reasons over user queries, selects appropriate tools, and refines its actions to provide grounded, up-to-date responses in natural language across multi-turn interactions. Evaluation across three phases shows that DeepSeek (thinking mode) excels in planning quality while Gemini 3.0 Flash delivers the highest overall performance and expert ratings, demonstrating the promise of agentic AI for improving clinical trial data access.