The paper introduces Confidence-Gated Transductive Test Generation (CoTT), a method that combines an efficient inductive approach with selective transductive generation triggered only when inductive confidence is low. This adaptive strategy enhances the reliability of test case outputs while limiting additional computational effort. Experiments on code reranking benchmarks demonstrate that CoTT surpasses previous baselines in performance metrics and reduces overall cost compared to applying transductive generation to every input.
arXiv:2505. 13553v3 Announce Type: replace-cross Abstract: The hallucination of code generation models hinders their applicability to systems requiring higher safety standards.
The paper introduces RobustTests, a framework that improves reinforcement learning for code generation by synthesizing test cases from faulty code and refining rewards with a dense, stepwise function. It uses validator agents and behavioral clustering to filter out invalid or redundant tests, and incorporates pass‑rate‑based rewards to counter hallucination noise. Experiments on CodeContests and LiveCodeBench show that fine‑tuning Qwen3‑32B with RobustTests yields a 3% absolute performance gain over baseline methods.
arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.
arXiv:2608. 04439v1 Announce Type: cross Abstract: Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations.
arXiv:2606. 00920v1 Announce Type: cross Abstract: Run-level pass rate overstates retry-free coverage by up to 17.
arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
HERO is a method that improves large language models for program generation by combining offline and test‑time harness optimization. It learns reusable prompt updates from execution‑verified successes and failures, validates candidate revisions before adoption, and then refines each generated code instance using execution feedback. On four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, raising pass@1 by 7.6 points over the initial prompt and adding another 2.0 points through test‑time refinement while improving execution efficiency.
arXiv:2512. 03086v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown remarkable capabilities in code translation, yet their performance deteriorates in low-resource programming domains such as Fortran and emerging frameworks like CUDA, where high-quality parallel data are scarce.
arXiv:2505. 03818v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics.
arXiv:2607. 02854v1 Announce Type: cross Abstract: Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT).
arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.