The paper introduces Test Cases Scaling (TCS), a two‑stage reinforcement learning framework that automatically generates sound and discriminative test cases for code‑generation large language models. Stage 1 trains a test generator to produce tests aligned with a reference solution, while Stage 2 refines the generator to produce counterexample tests that target the model’s current failure modes. Experiments on TACO and LiveCodeBench show that TCS improves pass@1 and inference‑time answer selection, and the learned generator also aids in selecting better outputs from other LLMs.
The paper introduces RobustTests, a framework that improves reinforcement learning for code generation by synthesizing test cases from faulty code and refining rewards with a dense, stepwise function. It uses validator agents and behavioral clustering to filter out invalid or redundant tests, and incorporates pass‑rate‑based rewards to counter hallucination noise. Experiments on CodeContests and LiveCodeBench show that fine‑tuning Qwen3‑32B with RobustTests yields a 3% absolute performance gain over baseline methods.
By Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
arXiv:2602. 20213v2 Announce Type: replace-cross Abstract: The evaluation of Large Language Models (LLMs) for code generation relies heavily on the quality and robustness of test cases.
By Jingwei Shi, Xinxiang Yin, Jing Huang, Jinman Zhao, Shengyu Tao
arXiv:2508.14313v4 Announce Type: replace-cross
Abstract: Test-time scaling strategies for Large Language Models predominantly rely on either reinforcement learning with sparse outcome rewards or sea...
By Can Jin, Yang Zhou, Qixin Zhang, Hongwu Peng, Di Zhang, Zihan Dong, Marco Pavone, Ligong Han, Zhang-Wei Hong, Tong Che, Dimitris N. Metaxas
The paper introduces X-Coder, a competitive programming model trained entirely on synthetic tasks, verified solutions, and reliable test cases, eliminating the need for real-world data in post‑training. A dual‑verification strategy is used to reduce noise in solutions and test outputs, providing high‑quality reward signals for reinforcement learning. X‑Coder‑14B achieves significant performance gains, scoring 67.5% on LiveCodeBench v5 and 63.4% on v6, surpassing its base model by over 40 points.
By Jie Wu, Haoling Li, Xin Zhang, Jiani Guo, Jane Luo, Xuewei Yang, Steven Liu, Yangyu Huang, Ruihang Chu, Scarlett Li, Yujiu Yang
arXiv:2607. 01181v1 Announce Type: cross Abstract: RL with verifiable rewards (RLVR) has emerged as a powerful paradigm for training LMs on tasks with well-defined success metrics, such as code generation and mathematical reasoning.
By Mehul Damani, Isha Puri, Idan Shenfeld, Jacob Andreas
arXiv:2606. 04310v1 Announce Type: new Abstract: Deep Neural Networks (DNNs) are increasingly being deployed in security-critical and safety-sensitive applications, which makes rigorous testing essential to identify and mitigate model weaknesses.
By Bin Duan, Matthew B. Dwyer, Guowei Yang
CovR is an agentic framework that automates testbench generation for hardware verification by combining self-reflection loops with simulation-based feedback to maximize coverage. It builds a large dataset of 16,514 specification–RTL reasoning tuples and uses reinforcement learning with tool-derived rewards to train a student model, achieving high coverage scores on VerilogEval, RTLLM V2.0, and CVDP. When deployed as a plug-in stimulus engine, CovR boosts coverage by nearly 19% and improves mutation detection while uncovering previously undetected failures.
By Manar Abdelatty, Maryam Nouh, Sherief Reda
The paper introduces Delta, a two‑phase framework for testing deep reinforcement learning agents. In the first phase, the agent under test is evaluated for catastrophic failures while collecting decision‑making data. The second phase trains a challenger agent from this data using offline RL; comparing the challenger’s rewards to the original agent reveals optimality bugs, and Delta successfully uncovered thousands of such issues across multiple environments.
By Junda He, Jieke Shi, Zhou Yang, Mingfei Cheng, David Lo
arXiv:2608. 12679v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in discovery domains such as math and science.
By Conor F. Hayes, Elliot Meyerson, Kajetan Schweighofer, Roberto Dailey, Babak Hodjat, Risto Miikkulainen, Xin Qiu
arXiv:2608. 10090v1 Announce Type: new Abstract: Large language models (LLMs) have advanced code generation, where executable feedback provides a more reliable learning signal than textual imitation alone.
By Hejia Zhang, Sheng Lu, Zhongming Yu, Chia-Tung Ho, Brucek Khailany, Jishen Zhao
TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.
By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song