arXiv:2606. 03070v1 Announce Type: cross Abstract: Asynchronous reinforcement learning can improve language-model post-training throughput by decoupling response generation from policy optimization, but stale responses introduce distribution drift.
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks.
arXiv:2607. 07508v1 Announce Type: cross Abstract: Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs).
arXiv:2607. 22186v2 Announce Type: replace Abstract: Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse.
arXiv:2606. 29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.
arXiv:2604. 00860v3 Announce Type: replace Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a central post-training paradigm for improving the reasoning capabilities of large language models.
arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
BRACE introduces an anchored Bellman‑residual correction to address stale critic bias in asynchronous reinforcement learning for language models. By limiting the correction horizon to a prefix of policy tokens and adding a constant‑weight Monte‑Carlo tail, it separates policy correction from reward propagation. The method improves mean@1 on BrowseComp‑Plus by 2.4% and runs 2.46× faster per step than synchronous training while staying stable 50 updates off‑policy.
The paper introduces Reverse‑Turn Policy Optimization (RTPO), a method that restructures multi‑turn agentic reinforcement learning rollouts into sparse reverse trees and updates policies in temporal reverse order. This approach addresses three key instability sources—context mismatch, weak turn‑level credit assignment, and asynchronous policy drift—by aligning each decision with its downstream continuation. Theoretical analysis shows RTPO eliminates context mismatch and drift, reduces credit bias, and converges to recursive optimality, while experiments demonstrate performance gains of 21.50% over trajectory‑level and 10.76% over turn‑level baselines on multi‑turn agentic RL benchmarks.
arXiv:2607. 06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs).
arXiv:2605.12070v3 Announce Type: replace-cross Abstract: Asynchronous reinforcement learning improves rollout throughput for large language model agents by decoupling sample generation from policy o...
Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on...
SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.