The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.
By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
arXiv:2610.09804v1 Announce Type: new
Abstract: Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimizatio...
By Yingxiang Yang, Weihang Xiao, Zhunxuan Wang, Joshua Flashner, Niresh Agarwal
Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient.
The paper introduces F-GRPO, a method that addresses the issue of reinforcement learning policies overfitting to common trajectories while neglecting rare correct ones. By deriving the probability of prompt‑local tail‑miss events and proposing a difficulty‑aware scaling coefficient inspired by Focal loss, the authors show that down‑weighting high‑success sampled groups can improve performance. Experiments on categorical simulations, Maze, and large language models (Qwen2.5‑7B) demonstrate that F‑GRPO raises average math pass rates and out‑of‑distribution performance without increasing group size or computational cost.
By Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov
arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.
By Ken Ding