arXiv Machine Learning
Oct 22

Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

Flat-RLVR introduces a sharpness-aware approach to reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). By targeting flat reward regions, the method reduces distribution sharpening, thereby preserving sample diversity while maintaining high single-attempt accuracy. The technique combines flat reward optimization with group-relative policy optimization (GRPO) variants to improve the trade-off between sampling efficiency and reasoning capacity.

By Yeonbum Kim, Hyun Kyu Lee, Sung Whan Yoon
arXiv Computation and Language
Aug 28

Boosting LLM Exploration via Weak-Model Guidance in RLVR

The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.

By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
arXiv Machine Learning
Jun 2

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

arXiv:2605. 21125v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs).

By Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu