arXiv:2608. 03222v1 Announce Type: cross Abstract: Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates.
By Chenyu Wang, Yunbo Lyu, Junda He, Zhou Yang, Chenxing Zhong, Yaniv Harel, David Lo
arXiv:2610.17633v1 Announce Type: new
Abstract: Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether su...
By Bingyu Shen, Jiaqi Tang, Lan Wei, Yi Sheng, J. Jenny Li, Boyang Li
Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion.
The paper investigates whether human or LLM‑judge‑identified ‘first mistake’ points truly indicate failure causes in LLM agents. By reconstructing intermediate states and computing a prefix value V(h_t) for each step, the authors identify a certified point of no return (PNR) and compare it to judge‑labeled steps. Their experiments on 150 SWE‑bench tasks show that judge labels often miss the decisive step, while restarting at the measured PNR improves success rates from 0.44 to 0.57.
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.
By Mohamed Abouzahra
arXiv:2607. 07405v1 Announce Type: new Abstract: Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully.
By Vikas Reddy, Sumanth Reddy Challaram, Abhishek Basu