arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
By Yu Li, Xiuyu Li, Mingyang Yi, Jiaxing Wang, zhangliangxu, Zhaolong Xing, Zhen Chen
The paper introduces a method called score centering to address the training‑inference mismatch (TIM) that destabilizes reinforcement learning for large language models. By adding an additive correction term that cancels drift between training and inference engines, score centering stabilizes RL and can match or surpass importance‑sampling techniques, especially as model size and mismatch severity increase. The approach also composes with importance sampling, yielding further performance gains in staleness experiments.
By Martin Marek, Max Ryabinin
Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away.
arXiv:2609.36816v1 Announce Type: new
Abstract: Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-poli...
By Ruichuan Huang, Jinghan Liu, Congliang Chen
arXiv:2602. 19327v3 Announce Type: replace Abstract: A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO).
By Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko
arXiv:2602. 04879v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm.
By Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee