arXiv Machine Learning By Zhengxin Zhang, Ning Wang, Sainyam Galhotra, Claire Cardie

How Far Are We From True Auto-Research?

Read the original on arXiv Machine Learning →

The paper "How Far Are We From True Auto-Research?" introduces ResearchArena, a lightweight scaffold that lets off‑the‑shelf agents (Claude Code, Codex, and Kimi Code) complete the full research cycle—ideation, experimentation, writing, and self‑refinement—across 13 computer science topics. In 39 trials, the agents produced 117 papers that were evaluated by three review methods: manuscript‑only review (SAR), artifact‑aware peer review (PR), and a human meta‑review. While SAR scores suggested competitive manuscript quality, PR revealed significant experimental shortcomings, with fabricated results, underpowered experiments, and plan/execution mismatches varying widely among agents; none of the papers met top‑tier venue acceptance thresholds, indicating a substantial gap to true auto‑research.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language
Sep 16

PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

PaperDoctor is an agent framework that provides evidence‑grounded, actionable feedback for scientific papers before submission. It evaluates writing, layout, references, code, theory, prior work, and experiments through a three‑layer hierarchical system, linking each critique to specific evidence and revision suggestions. The system selectively rebuilds and reruns experiments to uncover reproducibility gaps, and an interactive interface lets authors explore findings tied to their manuscript.

By Kevin Qinghong Lin, Siyuan Hu, Pan Lu, Yu Chen, Yanzhe Chen, Owen Queen, Yupeng Chen, Jialin Yu, Junchi Yu, Zifeng Ding, Yuanfeng Ji, Sheng Liu, Jindong Gu, Linjie Li, Mike Zheng Shou, Philip Torr, James Zou
arXiv Machine Learning
Jul 30

Can AI agents conduct open-ended AI research? Early evidence from two case studies

arXiv:2607. 27191v1 Announce Type: cross Abstract: Forecasts of explosive AI progress hinge on AI agents automating AI research.

By Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan
arXiv AI
Jul 21

AI for Auto-Research: Roadmap & User Guide

arXiv:2605. 18661v2 Announce Type: replace Abstract: AI-assisted research is crossing a threshold: fully automated systems can now generate research papers for as little as $15, while long-horizon agents can execute experiments, draft manuscripts, and simulate critique with minimal human input.

By Lingdong Kong, Xian Sun, Wei Chow, Linfeng Li, Kevin Qinghong Lin, Xuan Billy Zhang, Song Wang, Rong Li, Qing Wu, Wei Gao, Yingshuo Wang, Shaoyuan Xie, Jiachen Liu, Leigang Qu, Shijie Li, Lai Xing Ng, Benoit R. Cottereau, Ziwei Liu, Tat-Seng Chua, Wei Tsang Ooi
arXiv AI
Oct 23

CompLit: Scientific Literature Search Benchmarks Must Cover Implicit, Cumulative, and Unmet Needs

CompLit is a new literature search benchmark covering all eight arXiv domains, designed to evaluate agents on three diagnostic settings: implicit, cumulative, and unmet needs. The benchmark shows that top agents like Codex and Claude Code perform well on standard queries (86‑87%) but drop to 66‑68% on implicit queries, lose additional accuracy when requirements accumulate, and abstain on up to 53% of unmet queries. The study finds that agents often locate the correct or near‑correct paper but misjudge it, indicating a need for more reliable search capabilities.

By Dayoon Ko, Jihyuk Kim, Soyeong Jeong, Young-jun Lee, Dahyun Lee, Juyeon Kim, Gunhee Kim, Moontae Lee, Kyungjae Lee