arXiv:2610.29812v1 Announce Type: cross
Abstract: Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled...
By Mohammed Saqlain
The paper "How Far Are We From True Auto-Research?" introduces ResearchArena, a lightweight scaffold that lets off‑the‑shelf agents (Claude Code, Codex, and Kimi Code) complete the full research cycle—ideation, experimentation, writing, and self‑refinement—across 13 computer science topics. In 39 trials, the agents produced 117 papers that were evaluated by three review methods: manuscript‑only review (SAR), artifact‑aware peer review (PR), and a human meta‑review. While SAR scores suggested competitive manuscript quality, PR revealed significant experimental shortcomings, with fabricated results, underpowered experiments, and plan/execution mismatches varying widely among agents; none of the papers met top‑tier venue acceptance thresholds, indicating a substantial gap to true auto‑research.
By Zhengxin Zhang, Ning Wang, Sainyam Galhotra, Claire Cardie
arXiv:2610.19159v1 Announce Type: cross
Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper...
By Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck
The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
By Salman Rahman, Yubin Kim, Mihir Parmar, A. Ali Heydari, Genglin Liu, Simon A. Lee, Weizhi Zhang, Arian Hosseini, Ahmed A. Metwally, Yuzhe Yang, Baharan Mirzasoleiman, Xin Liu, Pavel Izmailov, Saadia Gabriel, Mark Malhotra, Shwetak Patel, Daniel McDuff, Hamid Palangi
The paper "Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers" introduces SciSlopBench, a dataset of 390 AI‑generated papers paired with human‑written counterparts, and defines six measures across Structure, Argument, and Artifacts to detect scientific slop. The authors show that these measures can identify AI papers with 85.9% accuracy and that higher slop correlates with lower ICLR ratings and distinguishes rejected from accepted papers. They also propose SciSlopHarness, a framework that guides a fixed LLM to revise only evidence‑supported sections, reducing the AI‑human gap by 63% without human reference targets.
By Yerim Oh, Young-Jun Lee, Jaewoo Ahn, Gunhee Kim, Dongyeop Kang
arXiv:2610.07937v1 Announce Type: new
Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context P...
By Filip Dorm, Leonora Vesterbacka