SWE-Shift is a new benchmark that tests coding agents in environments where tasks and workspaces can change mid-execution. It presents 120 scenarios across 12 change families, each evaluated at nine intervention points, comparing a Clean continuation (no change) with a Shifted continuation (task-relevant update). The benchmark shows that all evaluated agents perform better on Clean than on Shifted, with significant drops in success rates and zero paired success for certain change types.
arXiv:2607. 28591v1 Announce Type: cross Abstract: Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation.
Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification.
Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifa...
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on...
PAUSE is a steering framework for coding agents that intervenes once before the agent’s first source‑code edit, eliminating the need for learned monitors, checkpoint searches, or extra model calls. By having a Planner generate a repair plan from the task and pre‑edit transcript, the same agent can resume execution with this plan, preserving context and workspace. Experiments on SWE‑bench Verified and Multilingual show that PAUSE improves resolution rates by up to 16.8% online and recovers 35.4% of failed tasks offline, while reducing Executor token usage by 33‑41% compared to restarts.
The paper investigates how coding agents can self‑evolve their harnesses—software, configuration, and instructions—by reviewing completed sessions and proposing changes. Using Claude Code on 30 SWE‑bench issues, the authors analyzed 125 proposals from the etro skill, implemented 24, and evaluated 12 changes on subsequent tasks. The results show mixed effectiveness: some changes eliminated targeted problems, others reduced or had no impact, and some added unrelated work, indicating that self‑evolution requires mechanisms to assess and refine changes.
arXiv:2607. 15854v1 Announce Type: cross Abstract: Coding agents can fix a failing example without preserving the domain rule that made it fail, so later generations can repeat the same plausible mistake.
arXiv:2606.21804v2 Announce Type: replace-cross Abstract: Maintainability is a core dimension of software engineering, shaping how code is written, reviewed, and developed over time. While coding age...
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to the agents’ capabilities. In the arena, two agents take turns modifying a real repository: the Red agent adds a feature, introduces a subtle bug, and the Blue agent must fix it without knowing the bug exists. A tournament of 11 model‑harness participants on 15 repositories in five languages showed GPT‑5.5 as the best solver and Claude Opus 4.8 as the best task author, with newer model versions producing harder tasks for older ones.
arXiv:2608. 06701v1 Announce Type: cross Abstract: Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information needed to localize and repair it.