← Back to all news
arXiv AI

Function-Level Execution Feedback for Code Preference Optimization

The paper introduces STEP‑KTODER, a framework that applies process supervision to code generation by defining steps as module‑level functions in decomposed multi‑function programs and labeling each step with binary correctness derived from automatically generated unit tests. This approach combines function‑level supervision with outcome‑level feedback, improving preference optimization over outcome‑only methods such as KTO and DPO. Experiments on HumanEval, MBPP, BigCodeBench, and LiveCodeBench demonstrate that execution‑based labels are crucial, as LLM‑as‑a‑judge annotations tend to over‑predict failures and degrade downstream optimization.

Related stories

arXiv AI
3d ago

Ranking Function Generation with LLMs for Termination

The paper introduces RAINFALL, a neuro‑symbolic framework that generates ranking functions for proving loop termination by treating synthesis as a code‑completion task with placeholders and verifying the output with a symbolic checker. RAINFALL achieves a 90.22% success rate on 358 terminating C programs, outperforming existing baselines and boosting overall performance to 96.65% when combined with PROTON 2.1. The study also demonstrates the importance of placeholder contracts, few‑shot examples, symbolic verification, and diagnostic feedback across multiple LLMs.

By Rongkang Li, Jiaxin Hu, Taolue Chen, Yong Li, Fu Song, Zhilin Wu
llmsbenchmarks
More like this →
arXiv AI
Sep 2

Predicting Program Exit Code with LLMs and Programming Language Semantics

The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.

By Lara Marinov, Aditya Thimmaiah, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
llms
More like this →
arXiv AI
Oct 16

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.

By Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao
llmsbenchmarkssafety
More like this →
arXiv AI
Jun 8

SWE-IF: Aligning Code Evaluation with Human Preference

arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.

By Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun
llms
More like this →
arXiv Computation and Language
2d ago

ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generation

ReMCTS is a Monte Carlo Tree Search–style framework that enhances code generation by large language models (LLMs). It organizes program candidates as tree states, retains branch‑local debugging context, and retrieves failure experience across branches, distinguishing failed checks from unavailable evidence. Experiments on HumanEval and MBPP‑Sanitized show that visible‑test ReMCTS improves performance over direct generation in 8 of 10 model‑dataset pairs, while a pilot on HumanEval‑X C++ demonstrates compatibility with compiler‑backed execution.

By Huifei Wang, Xinying Huang, Yiheng Sun, Yifan Yuan
llmsbenchmarks
More like this →