The paper investigates what makes software issue resolution tasks difficult for agents by proposing a measurement framework and conducting a large‑scale empirical study on the CoderForge‑Preview dataset. It extracts static features from task patches, repositories, and prompts, and uses ensemble methods, SHAP attribution, and effect size analysis to predict task outcomes. The study finds that task difficulty is largely predictable from static features (AU C = 0.863), driven mainly by patch fragmentation and repository scale, with prompt linguistic features contributing for mid‑band tasks, suggesting a layered difficulty structure.
By Ebtesam Al-Haque, Brittany Johnson
arXiv:2608. 04682v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase.
By Haobin Li, Ping Deng, Weizhong Qian, Liang Jiang, Zhenyu Huang, Mouxing Yang, Xi Peng
The paper analyzes 2,487 tasks from five public coding‑agent benchmarks and 14,922 runs of six agent configurations, introducing the Spread‑Novelty‑Centrality (SNC) profile to describe how reference solutions are distributed, added, and the code’s dependencies, history, and complexity. It finds that every pair of benchmarks differs on at least two SNC axes, even when sharing the same task label, and that resolved runs tend to involve less distributed changes and lower centrality values, with patch sizes varying between Claude and Qwen configurations. The authors recommend that practitioners choose evaluation tasks based on change characteristics, inspect task descriptions, and assess patch scope alongside test outcomes to better align benchmarks with real‑world workloads.
By Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
The paper presents a systematic analysis of five state‑of‑the‑art automated program repair agents, tracing their decision‑making across 500 real‑world repair tasks. It finds that while the agents perform well on simple fixes, they struggle with logic‑intensive bugs, often producing verbose, overfitted patches that pass tests without addressing root causes. Key bottlenecks identified include poor test generation, limited regression test selection, and reliance on primitive tooling without access to debuggers or advanced program analysis tools.
By Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, Baishakhi Ray
arXiv:2604. 00594v2 Announce Type: replace Abstract: As the focus in LLM-based coding shifts from static single-step code generation to multi-step agentic interaction with tools and environments, understanding which tasks will challenge agents and why becomes increasingly difficult.
By Chris Ge, Daria Kryvosheieva, Daniel Fried, Uzay Girit, Kaivalya Hariharan
The paper introduces CTXGAP, a study measuring how much information coding agents omit from their user-facing reports compared to their full execution logs. Across 37 coding tasks, agents answer 93.6% of task-state questions when asked directly, but report-only readers recover only 52.0% of the same information, revealing a 40.7‑point gap. Even at final checkpoints of verified fixes, the gap remains significant, and detailed terminal scrollback improves recovery only to 73.6%. The study highlights that missing details in reports can leave supervisors unaware of failures, workarounds, and risks that need scrutiny.
By Guoyang Xu, Rui Zhang, Xiuyu Wu, Hao Chen
FeatureLoop is a benchmark that evaluates coding agents on 61 human‑reviewed tasks drawn from 34 repositories in five programming languages. Each task tracks a development goal through iterative feedback—adding or clarifying requirements and reporting failures—while cumulative executable checks monitor new behavior, repairs, and preservation of earlier functionality. In tests of 12 state‑of‑the‑art models, the best agent completed 72.1% of tasks with both feedback types, but only 15.4% of failures were recovered within two revisions, highlighting issues such as incomplete repairs, regressions, and token inefficiency.
By Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
By Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang, Xin Zhang, Minghua Ma, Pengfei Gao, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification.
SkillScriptBench is a 350‑task benchmark that evaluates how well large language models can self‑evolve executable agent skill packages by distinguishing between documentation repair, script repair, and preservation. The benchmark draws from 100 real GitHub skill packages, creating 150 repair tasks with injected script faults and a controlled track of 200 tasks that test clean, documentation‑faulty, script‑faulty, and combined‑faulty states. Experiments show that AST‑Guided Skill Revision—using abstract syntax trees to target relevant code locations—improves repair success by up to 27.7% and yields more consistent outcomes across repeated runs.
whyItMatters":"The benchmark provides a systematic way to measure and improve the reliability of automated skill package maintenance, which is essential for deploying robust LLM agents in real‑world applications."
By Yuxuan Liu, Haoran Li, Yuhao Zhang, Jiahe Guo, Hongyu Luo, Wenbin Hu, Huihao Jing, Kawai Chung, Junle Chen, Changxuan Fan, Qing Zong, Lingyun Xie, Yangqiu Song
Self-correction is particularly useful when a failure constrains the next repair. Coding agents benefit from this property because compilers, tests, and execution traces turn many failures into typed recovery signals, but broad language-agent tasks often expose only a coarse task failure.
arXiv:2606.21804v2 Announce Type: replace-cross
Abstract: Maintainability is a core dimension of software engineering, shaping how code is written, reviewed, and developed over time. While coding age...
By Shaswat Patel, Betty Li Hou, Arun Purohit, Kai Xu, Jane Pan, He He, Valerie Chen