The paper identifies a flaw in Group Relative Policy Optimization (GRPO) called the spurious advantage, where the advantage estimator can incorrectly reward rollouts that reach correct answers by guessing rather than reasoning. This issue appears in bounded-answer tasks, open-answer sets with bounded sub-cases, and search agents with many paths to the same answer. The authors propose SIGNBALANCE, a new magnitude estimator that removes composition, maintains verifier sign, applies a global scale, and uses stop‑gradient per‑class rescaling to restore zero‑mean balance, achieving comparable or better performance on math and search benchmarks.
arXiv:2610.09804v1 Announce Type: new
Abstract: Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimizatio...
By Yingxiang Yang, Weihang Xiao, Zhunxuan Wang, Joshua Flashner, Niresh Agarwal
The paper introduces Grounded Credit Assignment (GCA) for reinforcement learning in multi‑turn search agents. GCA separates credit for evidence retrieval and answer correctness, correcting the bias in GRPO that favors evidence‑retrieving rollouts only proportionally to accuracy gains. Applying GCA to Search‑R1 boosts average EM by 5.7–6.3 points and increases the proportion of evidence‑supported correct answers from 76.9% to 93.4%.
By Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo
The paper introduces F-GRPO, a method that addresses the issue of reinforcement learning policies overfitting to common trajectories while neglecting rare correct ones. By deriving the probability of prompt‑local tail‑miss events and proposing a difficulty‑aware scaling coefficient inspired by Focal loss, the authors show that down‑weighting high‑success sampled groups can improve performance. Experiments on categorical simulations, Maze, and large language models (Qwen2.5‑7B) demonstrate that F‑GRPO raises average math pass rates and out‑of‑distribution performance without increasing group size or computational cost.
By Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov
arXiv:2610.20497v1 Announce Type: new
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual...
By Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
The paper introduces Reinforcement Learning with Verifiable Rewards (RLVR) applied to small search agents, specifically training a Qwen3.5-0.8B model with Group Relative Policy Optimization and an interleaved Wikipedia-search tool on the MuSiQue dataset. Experiments varying reward shapes across three seeds show that RLVR can achieve a 3.8‑fold improvement over an untrained baseline, with the best run reaching a 0.352 average exact match. The study finds that the sparse exact‑match reward, standard in larger models, performs poorly for small models, indicating that reward design must be tailored rather than scaled down from large‑model recipes.
By Gaurisankar Jayadas, Aske Plaat, \'Alvaro Serra-G\'omez, Sandheep P