PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes non-degradation requirements, ownership, cross-file dependencies, and file-level constraints. The approach outperforms baselines on Paper2CodeBench, improving reference-based fidelity by 13.8% and halving high-severity evaluator critiques.
PaperCompiler is a framework that translates research papers into repository-level code by compiling paper-grounded evidence into explicit implementation specifications. It preserves source provenance, distinguishes between paper-supported, inferred, externally delegated, and unresolved information, and encodes requirements such as non-degradation, ownership, cross-file dependencies, and file-level constraints. The framework improves reference-based fidelity by 13.8% and cuts high-severity evaluator critiques from 13.2% to 6.1% on Paper2CodeBench.
By Yunhao Liu, Hong Phuc Pham, Jaehong Yoon
arXiv:2608. 19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions.
By Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
SWE‑bench Science is a benchmark for evaluating coding agents on scientific software engineering tasks, comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. The tasks are grouped into Issue‑driven, Expert‑exploratory, and Engineering‑integration paradigms, and even the best agent, Claude Code with Opus‑5 (max), achieves a pass@1 below 50%. The study identifies four common failure mechanisms—lack of scientific knowledge, misguided exploration, incomplete repair coverage, and poor generalization—and shows that providing scientific guidance can both help and hinder repair depending on its alignment with the task.
arXiv:2606. 18237v1 Announce Type: cross Abstract: Reproducing research results from papers and released code is central to scientific progress.
By Shanda Li, Qiuhong Anna Wei, Jingwu Tang, Valerie Chen, Nihar B Shah, Tim Dettmers, Yiming Yang, Ameet Talwalkar
arXiv:2608. 09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.
By Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, David Lo