IdeaAMBIG is a benchmark that evaluates how well research‑idea specifications are ready for implementation. It contains 660 evidence‑grounded instances, including 163 real‑world gaps from reproducibility reports and GitHub issues and 497 synthetic gaps added to codification‑ready references. The benchmark tests three capabilities—codification‑readiness assessment, defect localization, and clarification action generation—across 13 large language models, revealing that defect localization is the main bottleneck while clarification given the defect performs much better.
By Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
arXiv:2608. 09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.
By Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo
arXiv:2607. 28587v2 Announce Type: replace-cross Abstract: SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability.
By Manyi Wang, Junjielong Xu, Pinjia He
arXiv:2607. 08981v1 Announce Type: cross Abstract: LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed.
By Viraaji Mothukuri, Reza M. Parizi
The paper introduces MCR-Bench, a benchmark for realistic multi‑round code review that includes 2,269 real‑world tasks across five programming languages, each annotated with fine‑grained defect information and dynamic state labels. Experiments with mainstream large language models show limited overall performance, especially as interaction rounds increase, and reveal that model accuracy varies by defect type and severity. Error analysis identifies key failure mechanisms such as cross‑round temporal misalignment and insufficient long‑range memory.
By Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng
arXiv:2607. 00990v1 Announce Type: cross Abstract: Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories.
By Yaoqi Guo, Yang Liu, Jie M. Zhang, Yun Ma, Yiling Lou, Zhenpeng Chen
arXiv:2608. 14065v1 Announce Type: cross Abstract: Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques.
By Junchi Liu, Ali Bigdeli, Roya Daneshi, Atu Ambala, Sudipto Ghosh, Fabio Santos
arXiv:2608. 19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions.
By Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
arXiv:2507. 22580v2 Announce Type: replace-cross Abstract: Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention.
By Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, Eva Garcia-Lopez
arXiv:2608. 04783v1 Announce Type: cross Abstract: The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance.
By Yuexi Yang, Alyssa Wu, Ji Luo, Richeng Xuan, Zhichao Hu, Yuhong Liu, Zhen Qin
ReproAgent is a four‑stage pipeline—Prepare, Plan, Generate, Repair—that uses a persistent implementation contract to guide scientific AI agents in converting research papers into executable code repositories. The system employs two channels: an implementation‑requirement channel that translates paper snippets into code obligations, and a reference‑evidence channel that pulls content and structure from related repositories. Evaluated on PaperBench Code‑Dev, ReproAgent achieves the highest mean score among same‑backbone scaffolds for both Claude‑Sonnet‑4.5 and Gemini‑3‑Flash, with ablation studies confirming the contribution of both channels.
By Xue Hu, Zewei Pan, Zhongyuan Wang, Zhou Liu, Zeli Su, Wentao Zhang
arXiv:2605. 17561v2 Announce Type: replace-cross Abstract: Issues faced when using software are reported in the form of bug reports.
By Mahmut Furkan Gon, Emre Dinc, Tevfik Emre Sungur, Eray Tuzun