The paper addresses instability in off‑policy reinforcement learning for large language models (LLMs) caused by large policy lag. It shows that the correct correction for the prefix‑marginal policy gradient is the cumulative prefix importance ratio, which token‑level weighting fails to capture. To improve stability, the authors propose Minimum Prefix Ratio (MinPRO), a surrogate objective that uses the minimum token‑level ratio in the preceding prefix, and demonstrate its effectiveness on dense and mixture‑of‑experts LLMs across mathematical reasoning benchmarks.
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv:2609.36816v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-poli...
arXiv:2608. 01418v1 Announce Type: cross Abstract: Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models.
arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
EPIC introduces a new PPO-style clipping method that operates at the prefix level rather than token level, using the exact reverse-KL coordinate to regulate policy drift during autoregressive generation. By reformulating generation as a finite-horizon Markov decision process, the authors derive a performance-difference identity that identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each policy-improvement step. Experiments on off‑policy reinforcement learning with a sparse mixture‑of‑experts language model demonstrate that EPIC consistently outperforms standard and sequence‑level clipping baselines on mathematical‑reasoning benchmarks while remaining competitive with routing‑replay alternatives.
arXiv:2606. 01281v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs).
Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping...
arXiv:2606.22570v2 Announce Type: replace Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning ability of large language mo...
The paper introduces SORL, a framework that stabilizes off‑policy reinforcement learning for long‑horizon large language model agents. It identifies two key instability sources—token‑level policy granularity mismatches and high‑variance off‑policy updates—and proposes turn‑level importance sampling and clipping‑triggered normalization to align optimization with multi‑turn interactions. Two instantiations, SO‑PPO and SO‑GRPO, are evaluated on open‑domain, multi‑hop, and medical QA benchmarks, as well as on asynchronous RL for mathematical reasoning, showing improved robustness without the need for early stopping or heuristic tuning.
arXiv:2602. 19327v3 Announce Type: replace Abstract: A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO).