arXiv AI By Josip Vrdoljak, Akash Singh, Bas Steunebrink, Alex Yarotsky, John Lorenzo, Achille Nicoletti, Sam Pigott

ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases

Read the original on arXiv AI →

ExpertCrowd‑SWE (EC‑SWE) is a benchmark comprising 100 expert‑authored, multi‑file software engineering tasks designed to evaluate autonomous coding agents on large, unfamiliar codebases. Each task is a real change request from a practising engineer, paired with a sealed reference solution, a task‑specific grader that awards partial credit, and must‑pass blockers that mimic a maintainer’s merge criteria. The benchmark covers 87 open‑source projects across 11 languages, and the final 100 tasks were selected for their difficulty against current frontier models, with seven models evaluated over 3,495 runs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Oct 22

FrontierCS 2: A Benchmark for Computer Science R&D

FrontierCS 2 is a new benchmark comprising 48 open‑ended computer‑science research tasks that place language‑model agents inside a research environment, allowing them to submit incremental patches to a working repository. Each patch is scored by a judge on a development suite, while the final evaluation uses a held‑out suite and a private reference derived from the original authors’ code. The benchmark covers 13 domains, includes a private set of over 100 future tasks, and evaluates 12 agents, with Claude Opus 5.5 achieving the highest pass rate of 58%.

By Qiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang, Siddhant Ray, Bo Peng, Ruyi Ji, Shuo Yang, Haocheng Xi, Boru Chen, Xinyang David Han, Melissa Pan, Mayank Mishra, Boxi Yu, Sijia Liu, Kerem Akillioglu, Yi Pan, Zirui Wang, Mert Cemri, Diogo Antunes, Yichuan Wang, Aiwei Ivy Zhang, Zhenyang Chen, Qizheng Zhang, Kaiyuan Liu, Wenyuan Huang, Sherry Ruan, Shu Liu, Dacheng Li, Zerui Cheng, Zeyu Shen, Shang Zhou, Hanfei Chen, Yuzhong Zhang, Bohan Lyu, Qu Ao, Han Zheng, Chonghe Jiang, Jingzhuo Hu, Xiaokun Chen, Kunjun Li, Zhiqing Zhong, Haoye Tian, Tong Zhao, Yunzhong He, Hanchen Li, Ofir Press, Jiantao Jiao, Kunle Olukotun, Karthik R. Narasimhan, Jingbo Shang, Koushik Sen, Matei Zaharia, Alex Dimakis, Ion Stoica, Joseph E. Gonzalez, Alvin Cheung
arXiv AI
Oct 1

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limit...

By Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang
arXiv Computation and Language
Oct 12

SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/​Codex CLI as the top solver and Claude Opus 4.8/​mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.

By Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo