arXiv AI By Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan

Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

Read the original on arXiv AI →

The paper introduces MCPO (Margin-Credit Policy Optimization), a method that allocates reinforcement learning rewards to tokens where a language model hesitates, measured by the top‑2 margin of the next‑token distribution. MCPO redistributes each solution’s advantage across its tokens while keeping the mean unchanged, requiring no extra model evaluations. Experiments on MiniCPM5‑2B‑Base and Qwen3‑4B‑Base show MCPO consistently outperforms the baseline GRPO across multiple math benchmarks, especially on harder problems, and encourages longer, higher‑scoring answers.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
6d ago

Are Dense Rewards Worth Their Tokens in GRPO?

The paper evaluates whether dense, token‑level rewards improve training of language models compared to the standard group‑relative policy optimization (GRPO) algorithm. Using a matched‑compute protocol that equalizes generated tokens and GPU‑hours across interventions, the authors find that dense rewards—such as Monte‑Carlo step‑wise scoring—do not outperform GRPO, and in some cases perform worse even when matched for compute. Instead, a simple strategy of training on prompts predicted to be near the 50% success threshold yields consistent accuracy gains across multiple models and settings.

By Vikash Singh, Weicong Chen, Debargha Ganguly, Yanyan Zhang, Mohsen Hariri, Sreehari Sankar, Wang Yang, Chaoda Song, Alexander Nemecek, Osama Zafar, Yuting Shao, Kai Ye, Biyao Zhang, Xinpeng Li, Shouren Wang, Erman Ayday, Jing Ma, Sanmukh Rao Kuppannagari, Vipin Chaudhary
Hugging Face Trending Papers
Jul 30

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient.

arXiv AI
Sep 4

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

The paper introduces F-GRPO, a method that addresses the issue of reinforcement learning policies overfitting to common trajectories while neglecting rare correct ones. By deriving the probability of prompt‑local tail‑miss events and proposing a difficulty‑aware scaling coefficient inspired by Focal loss, the authors show that down‑weighting high‑success sampled groups can improve performance. Experiments on categorical simulations, Maze, and large language models (Qwen2.5‑7B) demonstrate that F‑GRPO raises average math pass rates and out‑of‑distribution performance without increasing group size or computational cost.

By Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daria Korotyshova, Daniil Gavrilov
arXiv Machine Learning
Jul 31

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.

By Ken Ding