The paper introduces SecPRBench, a benchmark that evaluates large language model agents on real-world bug‑introducing commits from critical software systems such as Firefox, V8, and OpenSSL. It automates harvesting, building, and validating these commits, and uses an LLM‑as‑a‑judge to match findings with ground‑truth vulnerabilities. Across eight state‑of‑the‑art models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT‑6 sol performing best, while security‑focused harnesses like OpenAI’s Codex Security and Anthropic’s Defending Code influence recall differently.
By Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang, Zhiheng Lyu, Lingming Zhang
arXiv:2608. 04682v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase.
By Haobin Li, Ping Deng, Weizhong Qian, Liang Jiang, Zhenyu Huang, Mouxing Yang, Xi Peng
arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.
By Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, Valentin Malykh
HAFixAgent is a history-aware program repair agent that incorporates blame-derived repository context into its repair loop. In a large-scale study on 854 Defects4J and 501 BugsInPy bugs, the agent’s best history heuristic improved repair rates by up to +64.3% with weaker models and outperformed existing agents by significant margins. The three complementary history heuristics together repair more bugs than any single heuristic, with only modest increases in agent steps or tokens.
By Yu Shi, Hao Li, Bram Adams, Ahmed E. Hassan
arXiv:2608.29675v1 Announce Type: cross
Abstract: Repository exploration is a distinct and costly stage of coding-agent pipelines: before generating a patch, an agent must identify which repository f...
By Mohammad Nour Al Awad, Sergey Ivanov
arXiv:2607. 25873v1 Announce Type: cross Abstract: Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent.
By Ramtin Ehsani, Irene Manotas, Saurabh Pujar, Luca Buratti, Preetha Chatterjee
arXiv:2609.23570v1 Announce Type: cross
Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluatio...
By Liyang Fan, Yingcheng Shi, Yongbin Li, Chenghao Sun, Xin Chen, Xander Xu, Hu Wei, Shiwen Ni, Min Yang, Jieping Ye
arXiv:2608. 14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.
By Yibo Yan, Huijuan Wang, Junzhou He, Yizhuo Liang, Shaoyu Wang, Huanchen Sun, Seo Jin Park
arXiv:2606.21804v2 Announce Type: replace-cross
Abstract: Maintainability is a core dimension of software engineering, shaping how code is written, reviewed, and developed over time. While coding age...
By Shaswat Patel, Betty Li Hou, Arun Purohit, Kai Xu, Jane Pan, He He, Valerie Chen
arXiv:2605. 17965v2 Announce Type: replace-cross Abstract: Bug localization remains a key bottleneck for large language model (LLM)-based software maintenance, where accurately identifying faulty code is essential for debugging, root cause analysis, triage, and automated program repair (APR).
By Md Afif Al Mamun, Gias Uddin
FeatureLoop is a benchmark that evaluates coding agents on 61 human‑reviewed tasks drawn from 34 repositories in five programming languages. Each task tracks a development goal through iterative feedback—adding or clarifying requirements and reporting failures—while cumulative executable checks monitor new behavior, repairs, and preservation of earlier functionality. In tests of 12 state‑of‑the‑art models, the best agent completed 72.1% of tasks with both feedback types, but only 15.4% of failures were recovered within two revisions, highlighting issues such as incomplete repairs, regressions, and token inefficiency.
By Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
By Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo