Switch‑SWE is a new benchmark that tests whether coding agents can resume an unfinished task after working on other tasks from the same repository. It provides 200 focus tasks from 19 Python repositories, each paired with matched uninterrupted and switched sessions, and evaluates agents on the final patch only. The benchmark reveals that interleaved work creates an average 11.5‑percentage‑point gap in task resolution, and a context‑management plug‑in can improve performance by 5.5 to 15.5 percentage points.
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
PAUSE is a steering framework for coding agents that intervenes once before the agent’s first source‑code edit, eliminating the need for learned monitors, checkpoint searches, or extra model calls. By having a Planner generate a repair plan from the task and pre‑edit transcript, the same agent can resume execution with this plan, preserving context and workspace. Experiments on SWE‑bench Verified and Multilingual show that PAUSE improves resolution rates by up to 16.8% online and recovers 35.4% of failed tasks offline, while reducing Executor token usage by 33‑41% compared to restarts.
Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifa...
arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...
The paper introduces CTXGAP, a study measuring how much information coding agents omit from their user-facing reports compared to their full execution logs. Across 37 coding tasks, agents answer 93.6% of task-state questions when asked directly, but report-only readers recover only 52.0% of the same information, revealing a 40.7‑point gap. Even at final checkpoints of verified fixes, the gap remains significant, and detailed terminal scrollback improves recovery only to 73.6%. The study highlights that missing details in reports can leave supervisors unaware of failures, workarounds, and risks that need scrutiny.
SWE-Shift is a new benchmark that tests coding agents in environments where tasks and workspaces can change mid-execution. It presents 120 scenarios across 12 change families, each evaluated at nine intervention points, comparing a Clean continuation (no change) with a Shifted continuation (task-relevant update). The benchmark shows that all evaluated agents perform better on Clean than on Shifted, with significant drops in success rates and zero paired success for certain change types.
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production S...
arXiv:2603.01209v3 Announce Type: replace Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python v...
arXiv:2610.14966v1 Announce Type: new Abstract: Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evalua...
Interlock is a framework that transforms long‑horizon tasks into typed completion graphs, linking evidence, decisions, and required effects through grounding, coupling, and realization edges. It compiles these graphs into a resettable workspace, a public request, and an evaluation rubric, and uses source‑coverage, public‑input, and agent‑as‑judge reviews to ensure that all mandatory requirements are met. Fine‑tuning Qwen3.6 backbones on Interlock‑generated trajectories improves pass rates on AutomationBench, JobBench, and Harvey LAB, showing that the synthesized experience benefits external task environments.
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv:2610.17757v1 Announce Type: new Abstract: Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incom...