arXiv:2607. 18163v1 Announce Type: cross Abstract: PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative.
arXiv:2607. 16244v1 Announce Type: cross Abstract: Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit.
arXiv:2601. 03895v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs).
arXiv:2606. 05434v1 Announce Type: new Abstract: Group Relative Policy Optimisation (GRPO) has emerged as an effective reinforcement-learning algorithm for aligning language models on reasoning tasks, but it treats every token position and every sampled rollout symmetrically.
arXiv:2609.09075v1 Announce Type: cross Abstract: In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term st...
arXiv:2606. 16733v1 Announce Type: new Abstract: Policy gradient algorithms for language models optimize the same objective $J(\theta) = \mathbb{E}*{\tau \sim p*\theta(\tau)}[R(\tau)]$, which has exactly two factors: the trajectory probability $p_\theta(\tau)$ and the reward $R(\tau)$.
Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient.
arXiv:2609.36802v1 Announce Type: new Abstract: A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning...
arXiv:2607. 27787v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability.
Group Adaptive Clipping Policy Optimization (GAPO) is a plug‑in modification to GRPO methods that adapts the importance‑sampling clipping boundary based on rollout advantage. By allowing rollouts with larger learning signals to receive proportionally greater update headroom, GAPO addresses the limitation of fixed clipping that suppresses rare but informative rollouts. Experiments on Qwen and Llama models show that GAPO consistently improves Pass@1 and Pass@k on math reasoning and coding benchmarks where base model pass rates are low.
arXiv:2608. 16072v1 Announce Type: cross Abstract: Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners.
arXiv:2608. 02181v1 Announce Type: new Abstract: Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets.
arXiv:2511. 02577v2 Announce Type: replace Abstract: Proximal Policy Optimization (PPO) is widely regarded as one of the most successful deep reinforcement learning algorithms, known for its robustness and effectiveness across a range of problems.