The paper introduces HSC-Bench, a benchmark with 1,039 tasks designed to evaluate how well receiving AI models can take over tasks after handoffs. It measures two key aspects: the ability to recover relevant repository evidence and the ability to continue from established conversational, tool, and file-work states. Experiments with six models show that structured handoff information consistently improves evidence recovery and, for most models, continuation performance, while the best performers differ between the two tracks.
By Wenyuan Li, Guang Li, Dongyuan Li, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
The paper introduces the evidence-action gap, where coding agents persist in tasks even when completion is impossible, despite having sufficient evidence that outside help is needed. To quantify this, the authors present ImpasseBench, a benchmark that injects realistic blockers into repair tasks and marks the step at which evidence becomes sufficient. Across eight models and 19,371 trajectories, they find that many agents still exhaust their budget or make unsafe requests after the evidence threshold, but an escalation tool and external monitor can significantly reduce these failures.
By Tianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou, Tongshuang Wu
arXiv:2608.24569v2 Announce Type: replace
Abstract: Large language model (LLM) agents build on one another's work through summaries, plans, and other intermediate artifacts. These records let a downs...
By Yiheng Sun, Huifei Wang, Yancheng Zhu, Zhenyu Li, Zebin Zhao, Yifan Yuan
arXiv:2606.28235v2 Announce Type: replace-cross
Abstract: In July 2026, about 1,200 AI agents in an internal OpenAI evaluation, meant to run in isolation, used a shared package server as a message bo...
By Daniel Russo
arXiv:2610.34167v1 Announce Type: new
Abstract: Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can comple...
By Jiaju Huang, Shaobin Chen, Xinglong Liang, Xinyu Ma, Hao Yang, Wei Ke, Tao Tan, Yue Sun
arXiv:2608. 16055v1 Announce Type: new Abstract: Existing agent benchmarks ask whether the agent finished the task.
By Bowen Li, Guojun Wang
The paper introduces SWE Refactor Bench, a benchmark of 20 whole‑repository migrations aimed at testing whether coding agents can autonomously perform long‑horizon stack migrations. It employs a three‑stage evaluation—Migration Audit, Behavioural Tests, and Agentic Verification—to assess both migration completeness and behavioural correctness, revealing that only 5.4 % of runs succeed across all stages. The study shows that agents struggle with migration tasks, especially language rewrites, and highlights the need for more robust coding‑agent capabilities.
By Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
arXiv:2603.01209v3 Announce Type: replace
Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python v...
By Victor May, Van Khue Nguyen, Aaditya Salgarkar, Yishan Wang, Diganta Misra, Huu Nguyen
PAUSE is a steering framework for coding agents that intervenes once before the agent’s first source‑code edit, eliminating the need for learned monitors, checkpoint searches, or extra model calls. By having a Planner generate a repair plan from the task and pre‑edit transcript, the same agent can resume execution with this plan, preserving context and workspace. Experiments on SWE‑bench Verified and Multilingual show that PAUSE improves resolution rates by up to 16.8% online and recovers 35.4% of failed tasks offline, while reducing Executor token usage by 33‑41% compared to restarts.
By Viral Patel, Boyin Tan, Jason Xue, Youcheng Sun
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release...
The paper introduces Banzhaf Resource Attribution (BRA), a factorial framework that quantifies how different inference-time resources—computation, information, and feedback—contribute to repairing failed coding agents. Experiments on six models across 194 function-level and 126 repository-level failures show that task-relevant specifications consistently improve repair, while public-test feedback offers modest gains and retrieved code offers little benefit. The study finds that as model capability increases, the value of extra computation diminishes, yet information remains useful, and that agents often prefer feedback even when it yields lower gains than random selection, revealing a mismatch between preferred and effective resources.
By Saiyue Lyu, Jerrold Huang, Yunbei Zhang, Janet Wang, Zhenting Wang, Yingqiang Ge, Haibo Zhang
arXiv:2609.13800v1 Announce Type: new
Abstract: Routing and cascades reduce tool-agent cost by transferring control across models, but stateful handoffs must preserve accepted choices, realized effec...
By Runzhi Deng, Yiming Zhong, Fang Zhao, Pan Zhou