← Back to all news
arXiv Machine Learning By Jinwoo Kim, Shraddha Barke

Understanding Enrichment in Reinforcement Learning

Read the original on arXiv Machine Learning →

The paper investigates how enrichment—adding intermediate guidance to sparse rewards in reinforcement learning—affects policy-gradient updates. It shows that omitting or truncating importance-weight corrections implicitly reweights rewards, and decomposes the resulting gradient error into scale, rotation, and variance components. The authors propose a sequential Monte Carlo weight correction that mitigates variance growth, and apply their analysis to a fine-tuned Qwen3-1.7B model on OpenMathReasoning, illustrating how enrichment helps prevent collapse in sparse domains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 3

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

The paper investigates why Group Relative Policy Optimization (GRPO) benefits from per‑prompt normalization by examining the local curvature of the sequence‑level policy gradient. It shows that standard deviation normalization acts as an adaptive gradient, yielding a provably faster convergence rate than unnormalized REINFORCE under mild conditions, with the improvement tied to the average within‑prompt reward standard deviation. The authors also propose IS‑GRPO, an importance‑sampling variant that maintains alignment with the full gradient and offers a tighter convergence guarantee, and empirically validate these theoretical insights on GSM8K and MATH datasets at 1.5B and 7B model scales.

By Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang
llmsreinforcement-learning
More like this →
arXiv Machine Learning
Oct 8

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

The paper introduces COPC, a Coupled Off-Policy Correction method for asynchronous reinforcement learning of large language models. COPC coordinates policy-side and advantage-side corrections by combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals, addressing both policy mismatch and advantage staleness. Experiments show COPC outperforms existing asynchronous baselines on tool-integrated mathematical reasoning and search tasks, while maintaining training stability and minimal overhead.

By Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen, Yuan Li, Qi Gu, Yan Feng, Hongyan Hao, Chao Qu
llmsreinforcement-learningsafety
More like this →
arXiv AI
Sep 17

Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

The paper identifies a failure mode in Proximal Policy Optimization (PPO) critics called Value Flattening, where state values vary sharply across intermediate states while critic predictions remain flat. The authors analyze this phenomenon theoretically and empirically, linking it to an implicit variance penalty and redundant updates from temporally correlated states. They propose Sparse Proximal Policy Optimization (SP³O), which applies the value loss to only a few well‑separated states, and demonstrate that this sparse supervision mitigates Value Flattening and improves policy performance on Qwen3‑Base across model sizes and evaluation suites.

By Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui, Ning Ding, Peilin Zhao, Yafu Li, Yu Cheng
llmsreinforcement-learning
More like this →