The paper introduces PolyHuman, a dataset of human-written programs in C++, Java, and Python, to test whether large language models can judge functional equivalence across languages. Using this dataset, the authors evaluate several open-weight and proprietary LLMs, finding that models struggle more with harder problems, show language-specific biases, and rely partly on superficial similarity cues. They also observe run‑to‑run instability in GPT‑o4‑mini, concluding that current LLMs do not reliably capture functional equivalence within or across programming languages.
By Hui Sun, Anderson Uch\^oa, Rohit Gheyi, Wesley K. G. Assun\c{c}\~ao
The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.
By Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
arXiv:2506. 02791v4 Announce Type: replace-cross Abstract: In recent years, code intelligence has gained increasing importance in the field of automated software engineering.
By Zhen Yang, Hongyi Lin, Yifan He, Junqi Wang, Zeyu Sun, Shuo Liu, Jie Xu, Pengpeng Wang, Zhongxing Yu, Qingyuan Liang
arXiv:2607. 27146v1 Announce Type: cross Abstract: Coding agents have made substantial progress on software engineering tasks that modify existing codebases, including bug fixing and feature implementation.
By Yihao Chen, Shi Chang, Khaled Chawa, Feng Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
arXiv:2606. 28998v1 Announce Type: cross Abstract: Large Language Model (LLM) alignment trains an LLM using preference data to produce outputs that better meet established quality standards.
By Gias Uddin, Sanjeepan Sivapiran
arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.
By Ankit Gupta, Aditya Prasad, Rameswar Panda
arXiv:2608. 02639v1 Announce Type: cross Abstract: Production prompts rarely carry a single instruction.
By Atul Anand, Sourav Chattaraj
arXiv:2607. 04631v1 Announce Type: new Abstract: The cost of producing code is rapidly diminishing with increasingly capable AI agents, while quality assurance of generated programs has not kept pace.
By Gabriel Poesia, Simon Henniger, Tzu-Han Hsu, Yilun Du, Nada Amin
arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.
By Sayed Erfan Arefin
arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
By Soham Dan
arXiv:2607. 13921v1 Announce Type: cross Abstract: Languages with rich static semantics, such as Rust, provide stronger guarantees for AI-generated code, but their strictness makes generation more difficult.
By Niels M\"undler-Sasahara, Hristo Venev, Dawn Song, Martin Vechev, Jingxuan He
arXiv:2507. 11687v5 Announce Type: replace-cross Abstract: Large language models excel at code generation but struggle with code linting, particularly in generalizing to unseen or evolving best practices beyond those observed during training.
By Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Yiqing Xie, Daniel Fried, Carolyn Rose