arXiv:2606. 07581v1 Announce Type: cross Abstract: A modern post-training pipeline often writes one symbol for its policy, pi_theta, while evaluating it through two different programs: a training kernel optimized for autograd and an inference kernel optimized for low-precision, fused, dynamically batched serving.
By Bruce Changlong Xu, Lan Wu
arXiv:2608.29296v1 Announce Type: cross
Abstract: Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this stati...
By Ziniu Li, Jinbo Wang, Guanhua Huang, Feiyuan Zhang, Pengbo Li, Alex Chen
ScaleRLT introduces a hybrid learner log‑probability method that reconciles differences between dense and deferred rollout computations in looped language models like Ouro. By using a stop‑gradient replay of the rollout schedule, the approach achieves bit‑for‑bit reproducibility across FP32 and bf16 real‑weight cells while maintaining the same gradient accuracy as dense implementations. The technique boosts rollout throughput by up to 1.56× on a single GPU and significantly speeds up multi‑node GRPO updates, leading to substantial accuracy gains on GSM8K in short training runs.
By Dezhen Lu, Zixiang Ni, Runguo Li
The paper presents an end‑to‑end MXFP8 reinforcement learning implementation on Blackwell GPUs, evaluating its benefits and limitations with Qwen3-8B and Qwen3-30B-A3B models. It finds that matching FP8 precision between sampler and learner improves consistency, that speed gains appear only for large micro‑batches, and that several weight‑synchronization correctness conditions can fail silently. The study also introduces seven explicit checks to detect such failures during training.
By Wengee Zhang
Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: training-inference divergence governs approximation error in finite-horizon bounds, whereas PPO clipping only gates sampled outward updates, acting as a sampled surrogate rather than a full-policy constraint.
arXiv:2607. 18722v1 Announce Type: new Abstract: Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing.
By Junyao Yang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Ruhan Wang, Xiangxin Zhou, Kishan Panaganti, Haitao Mi, Leowei Liang