arXiv AI By Jenny T. Liang, Mihika Bairathi, Wayne Chi, Ameet Talwalkar, Nishant Subramani, Valerie Chen

How Do Developers Edit AI-Generated Code, and How Well Do LLMs Infer Their Edits? Insights from 53.6K Developer Edits

Read the original on arXiv AI →

The paper introduces DECODE, a dataset of 53.6K real‑world in‑IDE edits of AI‑generated code from over 1,000 developers, and uses it to study when, why, and how developers modify such code. Findings show developers spend little effort editing and often abandon AI outputs, highlighting the need for LLMs to infer intent from edits. Experiments reveal that while frontier models can generate edits that align with developer intent, they tend to over‑edit and revert changes; fine‑tuning improves performance but does not fully close the gap.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Jul 7

A Preliminary Study on Explaining Risk of Code Changes using LLM-Based Prediction Models

arXiv:2607. 02782v1 Announce Type: cross Abstract: Predictions by machine learning (ML) and artificial intelligence (AI) models are often received skeptically unless they are paired with intelligible explanations.

By Yalin Liu, Kosay Jabre, Rui Abreu, Zachariah J. Carmichael, Vijayaraghavan Murali, Akshay Patel, Jun Ge, Weiyan Sun, Cong Zhang, Audris Mockus, David Khavari, Peter C. Rigby, Nachiappan Nagappan
arXiv AI
Oct 9

Code Understanding is a Bottleneck for Coding Agents

The paper introduces CABRA, a synthetic benchmark that constructs coding tasks from scratch using call‑graph transformations and scales difficulty along four axes: function traversal, search, runtime resolution, and instruction following. Experiments with eight large language models and six coding agents on 6,840 CABRA tasks show that while LLM accuracy degrades with task size, agents maintain high performance by leveraging tools; larger tasks trigger more tool calls, and a separate study on SWE‑bench verifies that tool‑call counts better predict agent accuracy than lines of code edited. The authors argue that CABRA exposes weaknesses in LLMs that are hidden by tool use and highlights the importance of code‑understanding abilities beyond mere editing. whyItMatters":"The study demonstrates that coding agents’ performance is more dependent on code‑understanding than on the amount of code edited, revealing a critical bottleneck for current coding agents."

By Nishant Balepur, Kiran Tomlinson, Tobias Schnabel
arXiv AI
Jun 9

Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks

arXiv:2606. 08676v1 Announce Type: cross Abstract: AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs).

By Shi Ying Chang, Chiok Yew Ho, Yichen Li, Yintong Huo