arXiv Machine Learning
Jun 9
Rethinking the Divergence Regularization in LLM RL
arXiv:2606. 09821v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a key component of post-training large language models (LLMs).
Rethinking the Trust Region in LLM Reinforcement Learning
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
Related stories
arXiv Machine Learning
Jul 14
Predictive Divergence Masks for LLM RL
arXiv:2607. 10848v1 Announce Type: new Abstract: Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates.
Hugging Face Trending Papers
Jul 12
Predictive Divergence Masks for LLM RL
Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away.
arXiv AI
Jun 29
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
arXiv:2512. 23075v5 Announce Type: replace-cross Abstract: Policy gradient methods for Large Language Models optimize a policy $\pi_\theta$ via a surrogate objective computed from samples of a rollout policy $\pi_{\text{roll}}$.
arXiv Machine Learning
Jun 30
The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.
arXiv Machine Learning
Jun 5
Extreme Region Policy Distillation
arXiv:2605. 25582v2 Announce Type: replace Abstract: Reinforcement learning for large language models faces a fundamental trade-off between sample efficiency and asymptotic performance: strictly on-policy methods discard trajectories after a single update, while off-policy reuse introduces distribution mismatch that existing trust-region techniques mitigate primarily by enforcing conservative optimization, often leaving rich training signals underutilized.
arXiv Machine Learning
Jun 2
Stabilizing Policy Optimization via Logits Convexity
arXiv:2603. 00963v2 Announce Type: replace Abstract: While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT).
arXiv Machine Learning
Jul 9
UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma
arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).
arXiv Machine Learning
Jun 9
Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
arXiv:2606. 08779v1 Announce Type: new Abstract: Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses.
arXiv AI
Jul 14
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
arXiv:2607. 10481v1 Announce Type: cross Abstract: Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile.
arXiv AI
Jun 30
Distributionally Robust Reinforcement Learning with Human Feedback
arXiv:2503. 00539v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs).
arXiv AI
Jul 7
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.