arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).
By Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu
The paper identifies a flaw in Group Relative Policy Optimization (GRPO) called the spurious advantage, where the advantage estimator can incorrectly reward rollouts that reach correct answers by guessing rather than reasoning. This issue appears in bounded-answer tasks, open-answer sets with bounded sub-cases, and search agents with many paths to the same answer. The authors propose SIGNBALANCE, a new magnitude estimator that removes composition, maintains verifier sign, applies a global scale, and uses stop‑gradient per‑class rescaling to restore zero‑mean balance, achieving comparable or better performance on math and search benchmarks.
arXiv:2606. 28707v1 Announce Type: new Abstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models.
By Yupeng Chang, Yuan Wu, Yi Chang
arXiv:2606. 24994v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for language-model reasoning can fail at both extremes of task difficulty: easy prompts often produce all-correct, low-diversity rollout groups with little gradient signal, while hard prompts can produce all-incorrect groups with no positive reward.
By Wenyang Hu, Junxiang Jia, Zhen Shu, Daniel Dahlmeier, See-Kiong Ng, Bryan Kian Hsiang Low
Group Relative Policy Optimization (GRPO) assigns a magnitude to each rollout based on within‑group reward statistics, rewarding rollouts that reach the correct answer through reasoning. However, the same magnitude can be high for rollouts that reach the answer by guessing, creating a spurious advantage that misleads the policy toward guess‑like behaviors. The paper identifies three scenarios where this occurs—bounded‑answer tasks, open‑answer sets with bounded sub‑cases, and search agents with many paths to the same answer—and proposes SIGNBALANCE, a composition‑free magnitude that preserves the verifier sign, uses a global scale, and restores zero‑mean balance via stop‑gradient per‑class rescaling, matching GRPO on open‑answer math and improving on bounded‑answer math and search agents.
By Jiamian Wang, Samyadeep Basu, Koustava Goswami, Tong Yu, Zhiqiang Tao
arXiv:2509. 22047v3 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available.
By Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Mitsuki Sakamoto, Ryota Mitsuhashi, Eiji Uchibe