arXiv AI

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

The paper investigates whether the importance ratio is essential for stable reinforcement learning (RL) in large language models (LLMs). It introduces Likelihood‑Gated Policy Optimization (LGPO), which removes the need to compute old‑policy likelihoods by using a likelihood‑based gating mechanism to enforce a soft trust region. Experiments show that eliminating the importance sampling correction does not hurt stability, while removing the trust region can cause collapse; LGPO matches or exceeds GRPO’s performance, reduces training time by about 18%, and remains stable in fully on‑policy settings where GRPO fails.

arXiv AI
Nov 5

Stabilizing Off-policy LLM Optimization with Prefix Importance Ratio

The paper addresses instability in off‑policy reinforcement learning for large language models (LLMs) caused by large policy lag. It shows that the correct correction for the prefix‑marginal policy gradient is the cumulative prefix importance ratio, which token‑level weighting fails to capture. To improve stability, the authors propose Minimum Prefix Ratio (MinPRO), a surrogate objective that uses the minimum token‑level ratio in the preceding prefix, and demonstrate its effectiveness on dense and mixture‑of‑experts LLMs across mathematical reasoning benchmarks.

By Shiye Lei, Zhihao Cheng, Dacheng Tao
arXiv Machine Learning
Sep 18

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.

By Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim, Joonseok Lee, Jaemoo Choi
Hugging Face Trending Papers
Jul 12

Predictive Divergence Masks for LLM RL

Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away.

Hugging Face Trending Papers
Jul 6

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.

arXiv Machine Learning
Jun 30

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.

By Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng
arXiv AI
6d ago

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

The paper introduces Stable Perturbation-Robust Policy Optimization (SPrPO), a reinforcement learning method that applies adaptive, sensitivity-aware perturbations during training to improve policy robustness against hidden‑state noise, pruning, and quantization. It defines a perturbation‑robust policy and analyzes conditions for stable monotonic improvement under perturbed updates. Experiments on ALFWorld and WebShop demonstrate that SPrPO enhances robustness across various perturbation types while preserving stable policy optimization.

By Pengxin Wang, Yuanzhe LI, Yuxin Ren, Huanrui Yang, Jingdi Chen