arXiv:2610.09633v1 Announce Type: new
Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
By Igor Slinko, Yaroslav Golubev, Sergey Titov
PAUSE is a steering framework for coding agents that intervenes once before the agent’s first source‑code edit, eliminating the need for learned monitors, checkpoint searches, or extra model calls. By having a Planner generate a repair plan from the task and pre‑edit transcript, the same agent can resume execution with this plan, preserving context and workspace. Experiments on SWE‑bench Verified and Multilingual show that PAUSE improves resolution rates by up to 16.8% online and recovers 35.4% of failed tasks offline, while reducing Executor token usage by 33‑41% compared to restarts.
By Viral Patel, Boyin Tan, Jason Xue, Youcheng Sun
Long-horizon coding requires agents to maintain an evolving view of the project they are changing: what problem is currently being solved, what has already been learned or modified, which other artifa...
arXiv:2610.14146v1 Announce Type: new
Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...
By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
The paper introduces CTXGAP, a study measuring how much information coding agents omit from their user-facing reports compared to their full execution logs. Across 37 coding tasks, agents answer 93.6% of task-state questions when asked directly, but report-only readers recover only 52.0% of the same information, revealing a 40.7‑point gap. Even at final checkpoints of verified fixes, the gap remains significant, and detailed terminal scrollback improves recovery only to 73.6%. The study highlights that missing details in reports can leave supervisors unaware of failures, workarounds, and risks that need scrutiny.
By Guoyang Xu, Rui Zhang, Xiuyu Wu, Hao Chen
SWE-Shift is a new benchmark that tests coding agents in environments where tasks and workspaces can change mid-execution. It presents 120 scenarios across 12 change families, each evaluated at nine intervention points, comparing a Clean continuation (no change) with a Shifted continuation (task-relevant update). The benchmark shows that all evaluated agents perform better on Clean than on Shifted, with significant drops in success rates and zero paired success for certain change types.
By Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo, Renrui Zhang, Hongsheng Li