The paper introduces CLOSEDLOOPGAUSSIAN, an adaptive curriculum strategy for Group Relative Policy Optimization (GRPO) that dynamically adjusts the pace and spread of an Easy-to-Hard Gaussian sampler based on probe estimates of per-difficulty accuracy. Unlike fixed easy-to-hard schedules, the method accelerates training when the model outpaces a reference curriculum, slows it down when it lags, and broadens sampling when earlier difficulties regress, thereby keeping training focused on learnable yet challenging tasks. Experiments on symbolic arithmetic reasoning demonstrate that CLOSEDLOOPGAUSSIAN outperforms vanilla GRPO and fixed open-loop curricula, especially in intermediate-difficulty regimes where group-relative learning signals are most informative.
By Yuv Raj Pant, Moshiur Rahman Tonmoy, Kaiqun Fu
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
By Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, Xiangyang Ji
arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).
By Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu
arXiv:2607. 26253v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal.
By Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes
The paper introduces Single‑Rollout Pass@k Policy Optimization (SR‑PPO), a reinforcement learning method that uses only one rollout per prompt to train language‑model agents. SR‑PPO combines Monte Carlo token‑level advantage estimation with Pass@k gradient reweighting and an online critic for baseline and weight estimation. Experiments show that SR‑PPO achieves stable learning and performance comparable to traditional Group Relative Policy Optimization (GRPO) on reasoning and logic benchmarks, even when off‑policy drift degrades standard Pass@1 PPO.
By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans
arXiv:2605. 12969v3 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks.
By Feng Zhang, Xinhong Ma, Ziqiang Dong, Xi Leng, Jianfei Zhao, Xin Sun, Yang Yang, Guanjun Jiang
RollVerify is a lightweight reinforcement learning framework that addresses the trade‑off between efficiency and accuracy in long‑tail rollout settings. It introduces an off‑policy shift metric (OPS) to quantify deviation in partially generated trajectories and uses sequence‑level and token‑level verification to truncate invalid suffixes before training. Experiments on mathematical reasoning tasks show that RollVerify matches on‑policy accuracy while cutting training costs, with preliminary evidence from code‑generation tasks.
By Yongqiang Yao, Jinru Tan, Kaihuan Liang, Zixin Yin, Yazhe Niu, Ruihao Gong, Dahua Lin, Ningyi Xu
arXiv:2508. 10123v3 Announce Type: replace-cross Abstract: Advanced reasoning in LLMs on challenging domains like mathematical reasoning can be tackled using verifiable rewards based reinforced fine-tuning (ReFT).
By Maxime Heuillet, Yufei Cui, Boxing Chen, Audrey Durand, Prasanna Parthasarathi
The paper introduces AdaRoll-GRPO, a method that allocates rollout budgets adaptively during Group Relative Policy Optimization (GRPO) training for large language models. By modeling rollout allocation as a Bayes‑adaptive Markov decision process, the approach learns an offline policy that assigns more rollouts to difficult questions and fewer to easier ones, all while keeping the total budget fixed. Experiments across several reasoning datasets and LLM architectures show that this adaptive strategy consistently improves GRPO training performance compared to the uniform allocation used in standard GRPO.
By Faeze Moradi Kalarde, Armin Toroghi, Nam Ho Koh, Doo Hwan Hwang, Seungjin Kim, Kee-Eung Kim, Scott Sanner
arXiv:2606. 05606v1 Announce Type: new Abstract: LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide.
By Yiming Zong, Yige Wang, Jiashuo Jiang
arXiv:2604. 00860v3 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models.
By Huaiyang Wang, Xiaojie Li, Deqing Wang, Haoyi Zhou, Zixuan Huang, Yaodong Yang, Jianxin Li, Yikun Ban
The paper introduces AnytimeReasoner, a framework that optimizes large language models for anytime reasoning by truncating the reasoning process to fit sampled token budgets and requiring the model to summarize the answer for verification. This approach adds verifiable dense rewards, improving credit assignment in reinforcement learning. The authors also propose Budget Relative Policy Optimization (BRPO), a variance reduction technique that enhances learning robustness and efficiency, and demonstrate that their method outperforms GRPO on mathematical reasoning tasks across various budgets and prior distributions.
By Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin