arXiv Machine Learning October 3, 2026 By Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

Read the original on arXiv Machine Learning →

The paper investigates why Group Relative Policy Optimization (GRPO) benefits from per‑prompt normalization by examining the local curvature of the sequence‑level policy gradient. It shows that standard deviation normalization acts as an adaptive gradient, yielding a provably faster convergence rate than unnormalized REINFORCE under mild conditions, with the improvement tied to the average within‑prompt reward standard deviation. The authors also propose IS‑GRPO, an importance‑sampling variant that maintains alignment with the full gradient and offers a tighter convergence guarantee, and empirically validate these theoretical insights on GSM8K and MATH datasets at 1.5B and 7B model scales.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Jul 25

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr.

llms reinforcement-learning safety

arXiv Machine Learning

Jul 28

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

arXiv:2607. 23364v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1.

By Fei Ding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Huiming Yang

llms reinforcement-learning safety

arXiv AI

Jun 30

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

arXiv:2606. 28707v1 Announce Type: new Abstract: Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms reinforcement-learning benchmarks

arXiv Machine Learning

Sep 23

PACT: From Credit Assignment to Critic Alignment

arXiv:2609.26355v1 Announce Type: new Abstract: Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted ma...

By Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan

llms agents reinforcement-learning efficiency benchmarks safety

arXiv AI

Jul 7

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

arXiv:2602. 18037v2 Announce Type: replace-cross Abstract: Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs).

By Johannes Ackermann, Michael Noukhovitch, Takashi Ishida, Masashi Sugiyama

llms reinforcement-learning safety

arXiv AI

Jul 2

GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity

arXiv:2607. 00152v1 Announce Type: cross Abstract: Three of the most popular methods for training language models to reason look like three different tricks.

By Yong Yi Bay, Kathleen A. Yearick

llms

One email a morning, machine-written

Related stories

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

PACT: From Credit Assignment to Critic Alignment

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity