arXiv:2606. 02875v1 Announce Type: new Abstract: Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue.
By Dipesh KC, Anjila Budathoki
The paper introduces the evidence-action gap, where coding agents persist in tasks even when completion is impossible, despite having sufficient evidence that outside help is needed. To quantify this, the authors present ImpasseBench, a benchmark that injects realistic blockers into repair tasks and marks the step at which evidence becomes sufficient. Across eight models and 19,371 trajectories, they find that many agents still exhaust their budget or make unsafe requests after the evidence threshold, but an escalation tool and external monitor can significantly reduce these failures.
By Tianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou, Tongshuang Wu
The paper introduces checkpoint handoff, an evaluation protocol that separates an agent’s ability to reach useful states from its ability to complete tasks in reinforcement learning. By using one checkpoint as a reacher up to a handoff point and another as a solver from the same replayed history, the authors can measure Reach (how often states within a fixed number of actions from success are achieved) and Solve (how often the task is completed from those states). Experiments on TravelPlanner and ALFWorld show that switching the solver from supervised fine‑tuning to RL yields larger gains when RL is used as the reacher, indicating that RL more effectively finds solvable states.
By Xuan Liu, Jingbin Qian
arXiv:2610.15407v1 Announce Type: new
Abstract: An end-to-end success rate alone cannot distinguish failure to learn an action from failure to apply it as task state changes. We study this distinctio...
By Dong Nie, Rui Lan, Yixin Nie, Lingzi Hong
arXiv:2608.24569v2 Announce Type: replace
Abstract: Large language model (LLM) agents build on one another's work through summaries, plans, and other intermediate artifacts. These records let a downs...
By Yiheng Sun, Huifei Wang, Yancheng Zhu, Zhenyu Li, Zebin Zhao, Yifan Yuan
The paper introduces an evidence‑guided recovery framework for AI agents that encounter ambiguous tool outcomes and partial execution, such as timeouts or failures. It defines a shared admissibility check for key expiry, authorization timing, dependencies, and residual effects, and a terminal certificate to distinguish actual goal satisfaction from publicly supported safe completion. In a finite study of 37 assignments, the approach achieved 28 certified completions, and in a larger comparison of 58 paired units, evidence‑guided advice certified 30 completions versus 9 for evidence prompting, demonstrating a significant improvement in task completion rates.
By Genliang Zhu, Xiaobo Chen, Ziyuan Wang, Chu Wang