← Back to all news
arXiv Machine Learning

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

CodeLift is a reinforcement learning approach that transforms a software engineering test suite into an adaptive curriculum by dynamically calibrating pass‑rate rewards. It groups tests that kill the same patches and those that pass the same patches, then prioritizes test groups that fewer patch groups pass, thereby reducing the influence of trivial tests. Experiments on SWE benchmarks demonstrate that CodeLift consistently improves pass@1/2/4/8 metrics compared to binary and pass‑rate baselines, providing denser and more stable learning signals.

Related stories

arXiv AI
Oct 15

SWEPatchTest: Evaluating Software Tests Beyond Reproduction

SWEPatchTest is a benchmark that evaluates software tests by measuring their ability to reproduce issues, accept correct patches, and reject incorrect ones across 619 tasks and 82,579 candidate patches. It introduces the Joint Validation Rate metric, revealing that even the best configurations only achieve 9.21% success when all three criteria are met. Using these validation scores to select demonstrations for supervised fine‑tuning can roughly triple the Joint Validation Rate and enhance downstream repair performance on multiple SWE‑bench datasets.

By Hua Lu, Pengcheng Wang, Han Li, Zichen Tian, Yanyu Chen, Huang He, Bingjin Chen
agentsfine-tuningbenchmarks
More like this →
arXiv AI
Oct 9

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.

By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
llmsagentsbenchmarks
More like this →
arXiv AI
Aug 26

Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

The paper introduces RobustTests, a framework that improves reinforcement learning for code generation by synthesizing test cases from faulty code and refining rewards with a dense, stepwise function. It uses validator agents and behavioral clustering to filter out invalid or redundant tests, and incorporates pass‑rate‑based rewards to counter hallucination noise. Experiments on CodeContests and LiveCodeBench show that fine‑tuning Qwen3‑32B with RobustTests yields a 3% absolute performance gain over baseline methods.

By Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
llmsagentsreinforcement-learningfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Sep 1

X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests

The paper introduces X-Coder, a competitive programming model trained entirely on synthetic tasks, verified solutions, and reliable test cases, eliminating the need for real-world data in post‑training. A dual‑verification strategy is used to reduce noise in solutions and test outputs, providing high‑quality reward signals for reinforcement learning. X‑Coder‑14B achieves significant performance gains, scoring 67.5% on LiveCodeBench v5 and 63.4% on v6, surpassing its base model by over 40 points.

By Jie Wu, Haoling Li, Xin Zhang, Jiani Guo, Jane Luo, Xuewei Yang, Steven Liu, Yangyu Huang, Ruihang Chu, Scarlett Li, Yujiu Yang
llmsreinforcement-learningfine-tuningbenchmarks
More like this →