The paper analyzes 2,487 tasks from five public coding‑agent benchmarks and 14,922 runs of six agent configurations, introducing the Spread‑Novelty‑Centrality (SNC) profile to describe how reference solutions are distributed, added, and the code’s dependencies, history, and complexity. It finds that every pair of benchmarks differs on at least two SNC axes, even when sharing the same task label, and that resolved runs tend to involve less distributed changes and lower centrality values, with patch sizes varying between Claude and Qwen configurations. The authors recommend that practitioners choose evaluation tasks based on change characteristics, inspect task descriptions, and assess patch scope alongside test outcomes to better align benchmarks with real‑world workloads.
arXiv:2609.01271v1 Announce Type: cross Abstract: Agentic software engineering benchmarks are typically summarized by nominal category labels such as "bug fix" or "feature implementation," yet benchm...
Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification.
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
arXiv:2606.21804v2 Announce Type: replace-cross Abstract: Maintainability is a core dimension of software engineering, shaping how code is written, reviewed, and developed over time. While coding age...
The paper investigates what makes software issue resolution tasks difficult for agents by proposing a measurement framework and conducting a large‑scale empirical study on the CoderForge‑Preview dataset. It extracts static features from task patches, repositories, and prompts, and uses ensemble methods, SHAP attribution, and effect size analysis to predict task outcomes. The study finds that task difficulty is largely predictable from static features (AU C = 0.863), driven mainly by patch fragmentation and repository scale, with prompt linguistic features contributing for mid‑band tasks, suggesting a layered difficulty structure.
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production S...
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifa...
arXiv:2605.29442v2 Announce Type: replace-cross Abstract: AI coding agents increasingly act directly within software environments, yet existing analyses of their failures rely on benchmark trajectori...
Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on...
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
The paper investigates how coding agents can self‑evolve their harnesses—software, configuration, and instructions—by reviewing completed sessions and proposing changes. Using Claude Code on 30 SWE‑bench issues, the authors analyzed 125 proposals from the etro skill, implemented 24, and evaluated 12 changes on subsequent tasks. The results show mixed effectiveness: some changes eliminated targeted problems, others reduced or had no impact, and some added unrelated work, indicating that self‑evolution requires mechanisms to assess and refine changes.