The study investigates how natural-language comments influence code generation by large language models. Observational and controlled experiments on LiveCodeBench reveal that comment frequency and general intent do not predict success, but comments derived from correct solutions significantly improve recipient model performance by an average of 17.2%. Conversely, comments from failed solutions or unrelated problems either offer no benefit or even reduce performance, and most models cannot fully recover the advantage of well‑crafted comments.
By Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, Xiaoning Du
arXiv:2601. 19072v3 Announce Type: replace-cross Abstract: Large Language models (LLMs) have shown strong capabilities in code review automation, such as review comment generation, yet they suffer from hallucinations -- where the generated review comments are ungrounded in the actual code -- poses a significant challenge to the adoption of LLMs in code review workflows.
By Kla Tantithamthavorn, Hong Yi Lin, Patanamon Thongtanunam, Wachiraphan Charoenwet, Minwoo Jeong, Ming Wu
The use of LLMs in software development has become increasingly widespread on tasks such as code generation and summarization. Reports from large technology companies showed that around 20% to 30% of their code are generated by LLMs.
arXiv:2606. 00049v1 Announce Type: cross Abstract: Large language models (LLMs) are widely recognised for their applications in natural language generation and are increasingly used for code generation tasks.
By Yuxi Chen, Yutian Tang, Timothy Storer
arXiv:2607. 10674v1 Announce Type: cross Abstract: As AI code tools become integrated into programming environments, students increasingly describe intended behavior in natural language and rely on these tools to generate code, shifting emphasis from code writing to specification.
By Nasser Giacaman, Valerio Terragni, Paul Denny, Viraj Kumar
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored.
arXiv:2501. 11086v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown tremendous promise in automated software engineering.
By Jing Liu, Seongmin Lee, Eleonora Losiouk, Marcel B\"ohme
arXiv:2607. 01867v1 Announce Type: cross Abstract: The use of LLMs in software development has become increasingly widespread on tasks such as code generation and summarization.
By Yongyi Ji, Jiaji Wang, Yi Zhou, Fuxiang Chen, Hongji Yang
arXiv:2608. 11513v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly integrated into software engineering workflows, helping developers write, debug, test, and maintain code.
By Alex Deaconu, Anubhav Gupta, Manaal Basha, Nicholas Haydu, Gema Rodr\'iguez-P\'erez
arXiv:2606. 24267v2 Announce Type: replace-cross Abstract: While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing.
By Hyunji Nam, Keertana Chidambaram, Dorottya Demszky, Natasha Jaques
arXiv:2507.21831v2 Announce Type: replace-cross
Abstract: LLMs are seeing widespread use for task automation, including automated coding in the social sciences. However, even though researchers have...
By Andreas Reich, Claudia Thoms, Tobias Schrimpf
arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.
By Saima Afrin, Alessandro Midolo, Camilo Escobar-Vel\'asquez, Mario Linares-V\'asquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo