arXiv AI

How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

arXiv:2607. 25873v1 Announce Type: cross Abstract: Large Language Model (LLM)-based Automated Program Repair systems are advancing rapidly, yet their performance remains inconsistent.

arXiv AI
Sep 1

Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study

The paper presents a systematic analysis of five state‑of‑the‑art automated program repair agents, tracing their decision‑making across 500 real‑world repair tasks. It finds that while the agents perform well on simple fixes, they struggle with logic‑intensive bugs, often producing verbose, overfitted patches that pass tests without addressing root causes. Key bottlenecks identified include poor test generation, limited regression test selection, and reliance on primitive tooling without access to debuggers or advanced program analysis tools.

By Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, Baishakhi Ray
arXiv AI
Sep 7

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

The paper investigates hallucination in large language model–based automated program repair (APR). It defines hallucination as producing patches or intermediate artifacts that are not grounded in available repair evidence, and analyzes it across final patches and intermediate tasks such as triggering test case identification, line coverage prediction, and additional test case generation. Experiments on 832 Defects4J bugs show that only 21.0%–55.9% of patches pass the developer test suite, with 72.7% of sampled repairs exhibiting hallucinations, often due to incorrect causal localization or repair strategies.

By Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
arXiv Computation and Language
Sep 25

Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?

The paper investigates how large language models (LLMs) handle bug fixing versus problem solving in competitive programming. Using a dataset of ~3,000 Codeforces submissions and their human fixes, the authors compare LLM-generated patches to human patches and assess whether LLMs prefer to modify buggy code or generate new solutions. Results show that LLMs often alter more lines than necessary and sometimes produce entirely new solutions, performing better when allowed to generate solutions from scratch rather than patching existing code.

By Alexandru Stefan Stoica, Traian Rebedea, Marian Cristian Mihaescu