The paper introduces Banzhaf Resource Attribution (BRA), a factorial framework that quantifies how different inference-time resources—computation, information, and feedback—contribute to repairing failed coding agents. Experiments on six models across 194 function-level and 126 repository-level failures show that task-relevant specifications consistently improve repair, while public-test feedback offers modest gains and retrieved code offers little benefit. The study finds that as model capability increases, the value of extra computation diminishes, yet information remains useful, and that agents often prefer feedback even when it yields lower gains than random selection, revealing a mismatch between preferred and effective resources.
arXiv:2609.15684v1 Announce Type: new Abstract: Language agents increasingly rely on reusable skills, but post-failure repair is often handled by opaque one-shot reflection: a model generates a skill...
arXiv:2608.25920v2 Announce Type: replace Abstract: As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerge...
arXiv:2610.41025v1 Announce Type: new Abstract: When a language-model agent fails a multi-step task, can another model find the decisive mistake and fix it with a single action? We introduce ReplayRC...
SWE‑Gate is a new repository‑level benchmark that evaluates software engineering agents on both functional correctness and review‑derived acceptance constraints. It creates 303 repair instances from real pull‑request review comments across 75 open‑source Python projects, providing separate functional and constraint tests along with compliant and non‑compliant patches. Experiments with four LLM backends show that while 644 repairs pass functional tests, 221 fail to meet the review constraints, highlighting a gap between functional success and full repair compliance.
PROBE is a benchmark that compiles 2,457 bugs fixed over 18 months in 20 GitHub repositories and provides a method to match candidate findings to this reference set. Evaluation of frontier coding agents shows they achieve less than 21% recall and under 9% precision, with stronger models improving recall from 8% to 17% but not precision. Despite low precision, 85% of findings are judged valid by LLM and human evaluators, and the distribution of bugs found by agents differs substantially from those fixed by maintainers.
HAFixAgent is a history-aware program repair agent that incorporates blame-derived repository context into its repair loop. In a large-scale study on 854 Defects4J and 501 BugsInPy bugs, the agent’s best history heuristic improved repair rates by up to +64.3% with weaker models and outperformed existing agents by significant margins. The three complementary history heuristics together repair more bugs than any single heuristic, with only modest increases in agent steps or tokens.
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release...
The paper introduces CTXGAP, a study measuring how much information coding agents omit from their user-facing reports compared to their full execution logs. Across 37 coding tasks, agents answer 93.6% of task-state questions when asked directly, but report-only readers recover only 52.0% of the same information, revealing a 40.7‑point gap. Even at final checkpoints of verified fixes, the gap remains significant, and detailed terminal scrollback improves recovery only to 73.6%. The study highlights that missing details in reports can leave supervisors unaware of failures, workarounds, and risks that need scrutiny.
Self-correction is particularly useful when a failure constrains the next repair. Coding agents benefit from this property because compilers, tests, and execution traces turn many failures into typed recovery signals, but broad language-agent tasks often expose only a coarse task failure.
arXiv:2606. 02875v1 Announce Type: new Abstract: Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue.
Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion.
arXiv:2611.08417v1 Announce Type: new Abstract: Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even be...