arXiv Computation and Language September 11, 2026 By Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin

Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

Read the original on arXiv Computation and Language →

The article introduces AgentActionBench, a benchmark designed to evaluate agent-based experiment reproduction across machine learning and AI4Science papers. It employs an MCP-based Action Recorder to capture agents’ behavior during reproduction and assesses the resulting traces against paper-specific rubrics. The benchmark includes 150 papers, with a human-annotated subset and model-assisted augmentation expanding it to over 10,000 rubric items, revealing that current systems face execution bottlenecks but that model-generated rubrics correlate strongly with human judgments.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jul 8

Prompt-to-Paper: Agentic AI System for Bioinformatics

arXiv:2607. 05456v1 Announce Type: new Abstract: While recent advances in large language models have enabled end-to-end automated manuscript generation, existing systems suffer from three critical deficiencies: (i) generated claims are not deterministically grounded in verifiable literature, (ii) experimental results are frequently fabricated rather than executed, and (iii) there exists no standardized, multi-dimensional framework to assess whether AI-generated manuscripts meet the quality and rigor required for real-world publication.

By Ramsha Kamran, Maheera Amjad, Zartasha Mustansar, Arsalan Shaukat, Salma Sherbaz, Muhammad U. S. Khan

llms rag agents safety

arXiv AI

Jun 29

Towards Automating Scientific Review with Google's Paper Assistant Tool

arXiv:2606. 28277v1 Announce Type: cross Abstract: Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving.

By Rajesh Jayaram, Drew Tyler, David Woodruff, Corinna Cortes, Yossi Matias, Vahab Mirrokni, Vincent Cohen-Addad

agents benchmarks

arXiv AI

Jun 2

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.

By Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, Xin Cong, Yankai Lin

llms agents benchmarks

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Jun 24

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

arXiv:2409. 11363v2 Announce Type: replace-cross Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research.

By Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, Arvind Narayanan

llms agents multimodal benchmarks

arXiv AI

Jun 2

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

arXiv:2507. 08038v3 Announce Type: replace-cross Abstract: Language model agents are increasingly used to automate scientific research, yet evaluating their scientific contributions remains a challenge.

By Talor Abramovich, Gal Chechik

llms agents benchmarks

One email a morning, machine-written

Related stories

Prompt-to-Paper: Agentic AI System for Bioinformatics

Towards Automating Scientific Review with Google's Paper Assistant Tool

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research