arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee
arXiv:2606. 09821v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a key component of post-training large language models (LLMs).
By Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, Tianyu Pang
The paper addresses instability in off‑policy reinforcement learning for large language models (LLMs) caused by large policy lag. It shows that the correct correction for the prefix‑marginal policy gradient is the cumulative prefix importance ratio, which token‑level weighting fails to capture. To improve stability, the authors propose Minimum Prefix Ratio (MinPRO), a surrogate objective that uses the minimum token‑level ratio in the preceding prefix, and demonstrate its effectiveness on dense and mixture‑of‑experts LLMs across mathematical reasoning benchmarks.
By Shiye Lei, Zhihao Cheng, Dacheng Tao
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
By Yu Li, Xiuyu Li, Mingyang Yi, Jiaxing Wang, zhangliangxu, Zhaolong Xing, Zhen Chen
arXiv:2609.36816v1 Announce Type: new
Abstract: Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-poli...
By Ruichuan Huang, Jinghan Liu, Congliang Chen
The paper introduces QUATRO, a reinforcement‑learning approach for fine‑tuning large language models that enforces trust‑region constraints directly rather than relying on heuristic clipping. By deriving a principled objective, QUATRO provides explicit control over policy updates and stabilizes entropy during training. Experiments on mathematical reasoning benchmarks demonstrate that QUATRO maintains stable training even with higher learning rates and increased policy staleness.
By Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Sookyung Kim, Joonseok Lee, Jaemoo Choi
Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv:2607. 10848v1 Announce Type: new Abstract: Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates.
By Xiangxin Zhou, Jiarui Yao, Penghui Qi, Bowen Ping, Jiaqi Tang, Haonan Wang, Tianyu Pang
arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.
By Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng
The paper introduces Stable Perturbation-Robust Policy Optimization (SPrPO), a reinforcement learning method that applies adaptive, sensitivity-aware perturbations during training to improve policy robustness against hidden‑state noise, pruning, and quantization. It defines a perturbation‑robust policy and analyzes conditions for stable monotonic improvement under perturbed updates. Experiments on ALFWorld and WebShop demonstrate that SPrPO enhances robustness across various perturbation types while preserving stable policy optimization.
By Pengxin Wang, Yuanzhe LI, Yuxin Ren, Huanrui Yang, Jingdi Chen
Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping...