← Back to all news
arXiv AI By Xiangkai Zeng, Zheng Liu, Zihao Lin

Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?

Read the original on arXiv AI →

The paper introduces Multi-step SWE, a benchmark consisting of 69 software engineering tasks that require a coding agent to perform 5–15 sequential, multi-hour engineering requests on a single codebase. Each step builds on the code produced in the previous step, making early design decisions and errors propagate throughout the chain. The benchmark covers four engineering work families, nine programming languages, and includes both from-scratch builds and existing open-source projects, with each step verified against a source of truth. Across nine state‑of‑the‑art models, the best achieves only an 18.5% solve rate, and 38 tasks remain unsolved by any model.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 1

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.

By Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 12

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.

By Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
agentsbenchmarks
More like this →
arXiv AI
Jun 9

SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?

arXiv:2606. 07682v1 Announce Type: cross Abstract: AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments.

By Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, Jiankai Sun, Yiyuan Li, Pramod Srinivasan, Ishan Gupta, Christopher Settles, Daniel Wang, Derek Chen, Pranav Raja, Albert Liu, Marek \v{S}uppa, Nevasini Sasikumar, Luyang Kong, Erik Quintanilla, Xiangyi Li, Ivan Bercovich, Steven Dillmann
agentsbenchmarkssafety
More like this →