← Back to all news
arXiv Computation and Language By Saiyue Lyu, Jerrold Huang, Yunbei Zhang, Janet Wang, Zhenting Wang, Yingqiang Ge, Haibo Zhang

Where Should a Stuck Agent Spend Its Next Tokens? Causal Attribution of Inference-Time Resources

Read the original on arXiv Computation and Language →

The paper introduces Banzhaf Resource Attribution (BRA), a factorial framework that quantifies how different inference-time resources—computation, information, and feedback—contribute to repairing failed coding agents. Experiments on six models across 194 function-level and 126 repository-level failures show that task-relevant specifications consistently improve repair, while public-test feedback offers modest gains and retrieved code offers little benefit. The study finds that as model capability increases, the value of extra computation diminishes, yet information remains useful, and that agents often prefer feedback even when it yields lower gains than random selection, revealing a mismatch between preferred and effective resources.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 4

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

SWE‑Gate is a new repository‑level benchmark that evaluates software engineering agents on both functional correctness and review‑derived acceptance constraints. It creates 303 repair instances from real pull‑request review comments across 75 open‑source Python projects, providing separate functional and constraint tests along with compliant and non‑compliant patches. Experiments with four LLM backends show that while 644 repairs pass functional tests, 221 fail to meet the review constraints, highlighting a gap between functional success and full repair compliance.

By Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 28

PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

PROBE is a benchmark that compiles 2,457 bugs fixed over 18 months in 20 GitHub repositories and provides a method to match candidate findings to this reference set. Evaluation of frontier coding agents shows they achieve less than 21% recall and under 9% precision, with stronger models improving recall from 8% to 17% but not precision. Despite low precision, 85% of findings are judged valid by LLM and human evaluators, and the distribution of bugs found by agents differs substantially from those fixed by maintainers.

By Marco Paina, Mark Niklas M\"uller, Niels M\"undler-Sasahara, Martin Vechev
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 12

HAFixAgent: History-Aware Agentic Program Repair

HAFixAgent is a history-aware program repair agent that incorporates blame-derived repository context into its repair loop. In a large-scale study on 854 Defects4J and 501 BugsInPy bugs, the agent’s best history heuristic improved repair rates by up to +64.3% with weaker models and outperformed existing agents by significant margins. The three complementary history heuristics together repair more bugs than any single heuristic, with only modest increases in agent steps or tokens.

By Yu Shi, Hao Li, Bram Adams, Ahmed E. Hassan
llmsagents
More like this →