E2E-SWE is a benchmark that tests large language models’ ability to create complete, functional software repositories from scratch. It includes 186 tasks across 11 programming languages, each requiring an agent to build an installable project based solely on a natural‑language specification and an empty workspace, while passing a hidden test suite. The benchmark was crafted by software engineers and LLMs, then refined through iterative verification by autonomous agents to ensure clarity and solvability.
By Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
By Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to the agents’ capabilities. In the arena, two agents take turns modifying a real repository: the Red agent adds a feature, introduces a subtle bug, and the Blue agent must fix it without knowing the bug exists. A tournament of 11 model‑harness participants on 15 repositories in five languages showed GPT‑5.5 as the best solver and Claude Opus 4.8 as the best task author, with newer model versions producing harder tasks for older ones.
arXiv:2609.38269v1 Announce Type: cross
Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limit...
By Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang
arXiv:2610.09079v1 Announce Type: cross
Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...
By Haibo Jin, Peng Kuang, Xucheng Yu, Jerry Wang, Dehao Wu, Haohan Wang
SWE‑Nexus is a fully automated pipeline that builds repository‑level implementation tasks by analyzing test execution. It uses four construction agents to select tests, trace production function bodies, and generate behavioral specifications, retaining scopes only when all expected scoring points pass and masking the traced bodies triggers failures. The benchmark covers 600 tasks from 105 repositories in six languages, and the best configuration resolves 37.8% of tasks in a single attempt, demonstrating the system’s ability to construct tasks from validated behavior beyond individual code changes.
By Songwei Yu, Ziqi Miao, Jingxuan Xu, Junhao Fan, Haoyang Huang, Yuan Xiong, Zihang Xu, Weihao Li, Wenqiang Zhu, Yuechen Luo, Xinshun Feng, Yanan Wu, Ken Deng