Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms, e.
LeanGRPO eliminates redundant recomputation in diffusion reinforcement learning by reusing computation graphs and activations from rollout for policy updates, or by backpropagating provisional gradients and correcting them later. It introduces two training schedules—LeanGRPO‑Retain and LeanGRPO‑Reweight—that target different model scales and input sizes. Experiments on FlowGRPO/DanceGRPO with FLUX.1‑dev and Wan show up to a 1.83× end‑to‑end speedup while preserving the original optimization objective.
By Sijie Wang, Zhiqiang Tan, Xinrui Yang, Shaohuai Shi
LeanGRPO eliminates redundant recomputation in diffusion reinforcement learning by reusing the same feed-forward backbone for rollout and policy update, thereby avoiding unnecessary gradient tracking. It introduces two training schedules—LeanGRPO‑Retain, which reuses computation graphs and activations, and LeanGRPO‑Reweight, which backpropagates provisional gradients and corrects them later. These methods achieve up to a 1.83× speedup on FlowGRPO/DanceGRPO with FLUX.1‑dev and Wan while preserving the original optimization objective.
arXiv:2604. 26256v2 Announce Type: replace Abstract: Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline.
By Tianhao Hu, Xiangcheng Liu, Yuchun Miao, Youshao Xiao, Hongyu Zang, Yang Zheng, Xuan Huang, Jinrui Ding, Yufei Zhang, Yu Yang, Yi-Kai Zhang, Yueqing Sun, Chengcheng Han, Xiandi Ma, Wei Wang, Qi Gu, Yerui Sun, Yuchen Xie, Xunliang Cai
arXiv:2606. 19004v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training of Diffusion Transformers (DiTs) is prohibitively expensive, requiring thousands of high-end GPUs.
By Ruiqi Lai, Dakai An, Wei Gao, Ju Huang, Siran Yang, Jiamang Wang, Lin Qu, Dmitrii Ustiugov, Wei Wang
arXiv:2607. 13394v1 Announce Type: cross Abstract: Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes.
By Xiaodong Liu, Michael Xu, Jack W. Stokes, Paul Smolensky, Doug Burger, Jianfeng Gao