The paper introduces RAINFALL, a neuro‑symbolic framework that generates ranking functions for proving loop termination by treating synthesis as a code‑completion task with placeholders and verifying the output with a symbolic checker. RAINFALL achieves a 90.22% success rate on 358 terminating C programs, outperforming existing baselines and boosting overall performance to 96.65% when combined with PROTON 2.1. The study also demonstrates the importance of placeholder contracts, few‑shot examples, symbolic verification, and diagnostic feedback across multiple LLMs.
By Rongkang Li, Jiaxin Hu, Taolue Chen, Yong Li, Fu Song, Zhilin Wu
The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.
By Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.
By Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao
arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.
By Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun
arXiv:2608. 04439v1 Announce Type: cross Abstract: Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations.
By Yiru Dong, Richong Zhang, Fanshuang Kong, Si Chen
arXiv:2610.15708v1 Announce Type: new
Abstract: Coding agents based on frozen language models can improve their performance by refining the external context they rely on, without updating model weigh...
By Yutong Cheng, Haifeng Chen, Peng Gao, Wei Cheng