Evaluating Harmful Memorization in LLM Code Generation

The paper introduces the Harmful Memorization Index (HMI) to measure when large language models (LLMs) fail to adapt to small semantic changes in coding tasks, producing code that is superficially similar to the original solution. It proposes an automated code‑rewriting pipeline that generates semantically different variants of tasks by altering core logic elements, and uses HMI to combine similarity metrics with performance degradation. Experiments on MBPP+ and BigCodeBench show that HMI does not simply grow with model size, is higher for certain logic‑edit types, and that supervised fine‑tuning can increase HMI while PPO offers a better accuracy–HMI trade‑off.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• reinforcement-learning
• fine-tuning