The paper investigates how coding agents can self‑evolve their harnesses—software, configuration, and instructions—by reviewing completed sessions and proposing changes. Using Claude Code on 30 SWE‑bench issues, the authors analyzed 125 proposals from the
etro skill, implemented 24, and evaluated 12 changes on subsequent tasks. The results show mixed effectiveness: some changes eliminated targeted problems, others reduced or had no impact, and some added unrelated work, indicating that self‑evolution requires mechanisms to assess and refine changes.
By Christoph Treude, Sebastian Baltes
Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification.
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
By Haomin Qi, Xingliang Wang, Xuanqi Gao, Baihui Sang, Xin Zhang, Minghua Ma, Pengfei Gao, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
The paper introduces the Self‑Evolving Agent Harness, a framework that allows an agent’s software scaffolding—prompts, memory, workflows, and executable components—to evolve while keeping the underlying model weights unchanged. It formalizes a feedback‑driven process where accepted revisions persist across runs, separating update targets from proposal and admission mechanisms, and distinguishes changes to interpreted state from revisions to tools and code. The authors analyze evolutionary search, provide generalization and retention bounds under PAC‑learning assumptions, and discuss the impact of persistent harness revisions on knowledge sharing, organizational workflows, and broader implications for scientific reliability, safety, and accountability.
The paper analyzes 2,487 tasks from five public coding‑agent benchmarks and 14,922 runs of six agent configurations, introducing the Spread‑Novelty‑Centrality (SNC) profile to describe how reference solutions are distributed, added, and the code’s dependencies, history, and complexity. It finds that every pair of benchmarks differs on at least two SNC axes, even when sharing the same task label, and that resolved runs tend to involve less distributed changes and lower centrality values, with patch sizes varying between Claude and Qwen configurations. The authors recommend that practitioners choose evaluation tasks based on change characteristics, inspect task descriptions, and assess patch scope alongside test outcomes to better align benchmarks with real‑world workloads.
By Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
SWE-Shift is a new benchmark that tests coding agents in environments where tasks and workspaces can change mid-execution. It presents 120 scenarios across 12 change families, each evaluated at nine intervention points, comparing a Clean continuation (no change) with a Shifted continuation (task-relevant update). The benchmark shows that all evaluated agents perform better on Clean than on Shifted, with significant drops in success rates and zero paired success for certain change types.
By Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo, Renrui Zhang, Hongsheng Li
arXiv:2610.38244v1 Announce Type: cross
Abstract: Existing code reveals past implementation decisions but rarely explains the reasoning behind them. When coding agents implement new features or modif...
By Haichao Zhu, Geyang Xu, Zhaorui Yang, Jiyuan Wang, Qian Zhang
arXiv:2610.18813v1 Announce Type: new
Abstract: Harness self-evolution revises an agent's software scaffolding, including prompts, memory, workflows, and executable components, while keeping model we...
By Chuyang Wei, Haoxiang Guan, Minghang Zhu, Kexin Zhou, Jiyan He, Xiren Zhou, Huanhuan Chen, Yifei Shen
arXiv:2603. 20667v2 Announce Type: replace-cross Abstract: Existing prompt-optimization techniques rely on local signals, causing poor generalization across tasks.
By Balaji Dinesh Gangireddi, Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan
Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifa...
arXiv:2607. 15854v1 Announce Type: cross Abstract: Coding agents can fix a failing example without preserving the domain rule that made it fail, so later generations can repeat the same plausible mistake.
By Muness Castle, Eric Rubeck
The paper introduces RuHarness, a framework that co‑evolves rubrics and harnesses to enable continual self‑improvement of autonomous agents that rely on large language models. RuHarness automatically discovers and reassesses behavioral criteria from execution trajectories, prioritizes unresolved weaknesses, and uses these insights to guide targeted harness updates while keeping the task objective unchanged. Experiments across varied agent tasks and backbone models demonstrate that RuHarness outperforms existing harness‑evolution baselines, especially on unseen held‑out tasks, and that its adaptive rubric prioritization supports sustained improvement and stable generalization over successive evolution rounds.
By Hao Chen, Yukun Yan, Ziyu Han, Ye He, Chuanzhe Guo, Shangzhan Li, Qi Shi, Qingfu Zhu, Maosong Sun, Wanxiang Che