arXiv AI

Stateless Language Agents: Scaling Long-Horizon Automated Research

arXiv AI
Jun 8

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.

By Jiayu Wang, Weijiang Lv, Bowen Fu, Jing Fu, Jiayi Song, Lingyu Zhang, Lanxuan Xue, Luodi Chen, Zepeng Xin, Kaiyu Li, Xiangyong Cao
arXiv AI
Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing
arXiv AI
Sep 25

IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents

IDRBench is a benchmark designed to evaluate the interactive capabilities of deep research agents that use large language models. It introduces controlled opportunities for clarification within a common workflow, comparing autonomous and interactive trajectories by measuring task‑specific report alignment and interaction cost. Experiments on 100 tasks with seven LLMs show that interaction consistently improves alignment, though its effectiveness varies depending on the agents’ questions and feedback integration.

By Yingchaojie Feng, Qiang Huang, Xiaoya Xie, Zhaorui Yang, Jun Yu, Wei Chen, Anthony K. H. Tung
arXiv AI
Jun 9

SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?

arXiv:2606. 07682v1 Announce Type: cross Abstract: AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments.

By Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, Jiankai Sun, Yiyuan Li, Pramod Srinivasan, Ishan Gupta, Christopher Settles, Daniel Wang, Derek Chen, Pranav Raja, Albert Liu, Marek \v{S}uppa, Nevasini Sasikumar, Luyang Kong, Erik Quintanilla, Xiangyi Li, Ivan Bercovich, Steven Dillmann
arXiv AI
23h ago

EPOCH: Reliable Discovery through Evidence-Governed Search

EPOCH is an evidence‑governed architecture for AI research agents that improves how candidate discoveries are evaluated and validated. By combining explicit task contracts, typed memory, active falsification, admission checks, and independent replay, EPOCH ensures that each candidate is judged against the strength and scope of its claim. The system achieves state‑of‑the‑art performance on several benchmarks, including AlgoTune, Math14, and AgentHPO, and demonstrates significant advances in executable constructions, algorithm optimization, counterexamples, and proof‑supported results across ten discovery problems.

By Binjie Guo, Aisheng Mo, Ruitong Li, Xinle Deng
arXiv Machine Learning
Sep 29

AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework

The paper reports on applying AutoResearch—a large language model that iteratively edits training scripts—to optimize embedding systems for a book recommendation pipeline at production scale. Over twelve weeks, the authors ran 220+ experiments across two representation‑learning systems, uncovering five recurring failure modes (infrastructure fragility, agent memory decay, search‑direction stagnation, iteration‑cost asymmetry, and metric fixation) that were not present in smaller settings. They propose a three‑principle scaffolding (prevent, persist, redirect) to address these failures, achieving a 1.82× lift in Recall@6, a 2.1× lift in coherence, and an autonomous text‑only fallback that expanded catalog coverage by 5.8×.

By Aparajith Chandran, Juwon Kim, Saurav Jha, Pablo Castells, Florian Hottier