← Back to all news
arXiv AI By Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang

Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations

Read the original on arXiv AI →

The study examines how large language models (LLMs) repair code after runtime or API migrations, revealing that many repaired procedures run without errors yet produce incorrect results. Across 73 tasks and 60 documented changes in CPython, Node.js, and SQLite, the authors found that typical test inputs allowed 41% of wrong repairs to pass, while targeted edge inputs reduced this to 6–7%. Even with additional hard distractors and blind model writers, a significant fraction of incorrect repairs still passed the preregistered tests.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 16

Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair

The paper "Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair" investigates why large language model (LLM) code‑repair agents sometimes produce incorrect patches. By analyzing 1,844 model requests on two synthetic Python tasks—PersistQueue and Dijkstra—the authors trace how misconceptions in a model’s hypothesis or implementation plan lead to specific software obligation violations. They find that shifting a misconception from hypothesis to plan increases failure rates by roughly 30% for both tasks, revealing distinct, model‑dependent failure patterns linked to upstream misunderstandings.

By Veljko Skarich
llmsagents
More like this →
arXiv AI
Sep 7

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

The paper investigates hallucination in large language model–based automated program repair (APR). It defines hallucination as producing patches or intermediate artifacts that are not grounded in available repair evidence, and analyzes it across final patches and intermediate tasks such as triggering test case identification, line coverage prediction, and additional test case generation. Experiments on 832 Defects4J bugs show that only 21.0%–55.9% of patches pass the developer test suite, with 72.7% of sampled repairs exhibiting hallucinations, often due to incorrect causal localization or repair strategies.

By Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
llmssafety
More like this →
arXiv AI
Sep 4

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

The paper reports a preregistered audit of language‑model judges used as measurement instruments, revealing that the assumption that a model’s responses remain stable over time is invalid. Across nearly 53,000 audited requests, repeat rankings and byte‑identical replays fell far below required reliability thresholds, with three identified mechanisms—label‑to‑meaning bias, candidate gaps below the noise floor, and input permutation noise—explaining the discrepancy. The study proposes a three‑level snapshot‑identity framework, eight design rules, and a reporting checklist to prevent such reliability failures in future evaluations.

By Haoyaun Zhu, Jie Zhang
llms
More like this →