The study examines how large language models (LLMs) repair code after runtime or API migrations, revealing that many repaired procedures run without errors yet produce incorrect results. Across 73 tasks and 60 documented changes in CPython, Node.js, and SQLite, the authors found that typical test inputs allowed 41% of wrong repairs to pass, while targeted edge inputs reduced this to 6–7%. Even with additional hard distractors and blind model writers, a significant fraction of incorrect repairs still passed the preregistered tests.
By Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang
The paper "Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair" investigates why large language model (LLM) code‑repair agents sometimes produce incorrect patches. By analyzing 1,844 model requests on two synthetic Python tasks—PersistQueue and Dijkstra—the authors trace how misconceptions in a model’s hypothesis or implementation plan lead to specific software obligation violations. They find that shifting a misconception from hypothesis to plan increases failure rates by roughly 30% for both tasks, revealing distinct, model‑dependent failure patterns linked to upstream misunderstandings.
By Veljko Skarich
The paper investigates hallucination in large language model–based automated program repair (APR). It defines hallucination as producing patches or intermediate artifacts that are not grounded in available repair evidence, and analyzes it across final patches and intermediate tasks such as triggering test case identification, line coverage prediction, and additional test case generation. Experiments on 832 Defects4J bugs show that only 21.0%–55.9% of patches pass the developer test suite, with 72.7% of sampled repairs exhibiting hallucinations, often due to incorrect causal localization or repair strategies.
By Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
The paper investigates whether large language models (LLMs) can automatically verify no-code bug fixes—solutions that instruct users to change settings, upgrade versions, or alter workflows—by executing them in a real browser environment. Using 322 fixes from a benchmark, the study tests three executor agents (OpenCUA‑72B, Claude Sonnet 5, and Meta’s Muse Glimmer) and finds that only 17.6% of issues could be set up and passed sanity checks, with resolution rates ranging from 14.6% to 49.7% depending on the executor. Even the best configuration resolved fewer than half of the fixes, and executor choice significantly affected outcomes, highlighting the need for execution-based verification before such fixes reach users.
By Utku Boran Torun, Veli Karakaya, Eray T\"uz\"un
arXiv:2609.05881v1 Announce Type: cross
Abstract: Developers increasingly run large language models locally by pulling quantized GGUF artifacts from public registries, yet nothing in the distribution...
By Aditi Patodiya
arXiv:2608.23001v1 Announce Type: new
Abstract: Automated manuscript pipelines often regenerate an entire section to repair a local defect, allowing unrelated metrics and citations to change even whe...
By Weiwei Yang
The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.
By Haoyaun Zhu, Jie Zhang
arXiv:2607. 06636v1 Announce Type: cross Abstract: Large language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and other specification-defined corner conditions.
By Amin Haeri, Mahdi Ghelichi
SkillScriptBench is a 350‑task benchmark that evaluates how well large language models can self‑evolve executable agent skill packages by distinguishing between documentation repair, script repair, and preservation. The benchmark draws from 100 real GitHub skill packages, creating 150 repair tasks with injected script faults and a controlled track of 200 tasks that test clean, documentation‑faulty, script‑faulty, and combined‑faulty states. Experiments show that AST‑Guided Skill Revision—using abstract syntax trees to target relevant code locations—improves repair success by up to 27.7% and yields more consistent outcomes across repeated runs.
whyItMatters":"The benchmark provides a systematic way to measure and improve the reliability of automated skill package maintenance, which is essential for deploying robust LLM agents in real‑world applications."
By Yuxuan Liu, Haoran Li, Yuhao Zhang, Jiahe Guo, Hongyu Luo, Wenbin Hu, Huihao Jing, Kawai Chung, Junle Chen, Changxuan Fan, Qing Zong, Lingyun Xie, Yangqiu Song
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
By Parker Fawcett
arXiv:2607. 28887v1 Announce Type: cross Abstract: Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain.
By Amir M. Ebrahimi, Mohammed Mehedi Hasan, Aaditya Bhatia, Gopi Krishnan Rajbahadur, Ahmed E. Hassan
The paper investigates how large language model (LLM) agents lose information when cutting tool output before it reaches the model. In a real Linux‑kernel‑trace agent, up to one‑third of tool results were truncated, producing broken JSON and unevenly deleting data near the end of the output. Experiments show that this truncation increases computational effort and reduces accuracy, though it is unclear whether the loss directly causes the drop in correctness.
By Yuvraj Sehgal, Mahsa Panahandeh, Naser Ezzati-Jivan