Li$^{2}$SAO is a lightweight variant of Single-rollout Asynchronous Optimization (SAO) that eliminates the need for a pretrained value model by using group-relative actor‑critic bootstrapping. The method first trains the actor with a group-relative advantage and the critic with a group-centered return before switching to single‑rollout Generalized Advantage Estimation. Experiments on single‑turn reasoning and multi‑turn agentic tasks show that Li$^{2}$SAO matches or surpasses the performance of GRPO, especially in early training stages.
By Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks.
arXiv:2607. 07508v1 Announce Type: cross Abstract: Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs).
By Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong
arXiv:2608. 19842v1 Announce Type: new Abstract: Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models.
By Dayang Liang, Lang Feng, Bo An, Yunlong Liu
Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks.
ThunderSyncRL is a training framework that eliminates idle time in agentic reinforcement learning by starting gradient computation immediately once all necessary inputs are available, thereby avoiding policy staleness. It applies to group relative policy optimization (GRPO) by computing trajectory score gradients as soon as rewards arrive, and to on‑policy distillation (OPD) by updating gradients for completed agentic turns while tool calls execute. Experiments on SWE‑bench Verified and Terminal Bench 4.0 show that ThunderSyncRL matches synchronous training performance up to 1.9× faster and outperforms asynchronous training by up to 2.47 percentage points at a fixed budget.
arXiv:2609.37119v1 Announce Type: cross
Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry
The paper presents an analysis of asynchronous reinforcement learning for large language model post‑training, focusing on the impact of stale rollouts. It derives a convergence bound for GRPO‑style algorithms that balances the gradient estimator’s second moment and bias, and introduces a new group‑mass capping GRPO (GMC‑GRPO) method that reduces delay‑dependent bias. Experiments on Qwen3 models and reasoning benchmarks show that GMC‑GRPO outperforms existing baselines in robustness to large rollout delays.
By Qijia He, Ruinan Jin, Jun Luo, Shaofeng Zou, Yingbin Liang
arXiv:2610.17666v1 Announce Type: cross
Abstract: Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps....
By Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin, Wei Lu
The paper investigates why Group Relative Policy Optimization (GRPO) benefits from per‑prompt normalization by examining the local curvature of the sequence‑level policy gradient. It shows that standard deviation normalization acts as an adaptive gradient, yielding a provably faster convergence rate than unnormalized REINFORCE under mild conditions, with the improvement tied to the average within‑prompt reward standard deviation. The authors also propose IS‑GRPO, an importance‑sampling variant that maintains alignment with the full gradient and offers a tighter convergence guarantee, and empirically validate these theoretical insights on GSM8K and MATH datasets at 1.5B and 7B model scales.
By Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang
The paper introduces Single‑Rollout Pass@k Policy Optimization (SR‑PPO), a reinforcement learning method that uses only one rollout per prompt to train language‑model agents. SR‑PPO combines Monte Carlo token‑level advantage estimation with Pass@k gradient reweighting and an online critic for baseline and weight estimation. Experiments show that SR‑PPO achieves stable learning and performance comparable to traditional Group Relative Policy Optimization (GRPO) on reasoning and logic benchmarks, even when off‑policy drift degrades standard Pass@1 PPO.
By Fengdi Che, Yang Liu, Lei Yu, Meng Cao, Tong Che, Rupam Mahmood, Dale Schuurmans
Group-based reinforcement learning (RL) methods, such as GRPO and its variants, have become a leading paradigm for training reasoning and agentic large language models (LLMs). While their group-normal...