arXiv Machine Learning

HARTS: A Prefix-Sharing Training System for Agentic RL with Hybrid-Attention Models

arXiv Machine Learning

Aug 31

HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees

HARTS (Hybrid‑Attention RL over Tree Structures) is a new system that jointly plans microbatches, data‑parallel replica assignments, and microbatch‑slot schedules to efficiently train agentic reinforcement learning models with hybrid attention over arbitrary rollout trees. It uses prefix compression and a linear‑time algorithm for chunkwise linear attention to avoid recomputing shared prefixes, enabling activation recomputation and bounded state replay while preserving trajectory‑wise training benefits. In experiments on an Agentic RL workload derived from SWE‑bench tasks, HARTS delivers 4.81–4.87× speedups in forward, backward, and gradient computations across multiple parallel configurations, with numerical differences comparable to baseline self‑rerun variation and a similar reward trend over the first 120 training steps.

By Boyuan Meng (Ant Group, China), Peihua Bao (Ant Group, China), Hong Liu (Ant Group, China), Xiaowei Zhu (Ant Group, China), Chao Wang (Ant Group, China), Gen Li (Ant Group, China), Zhenxuan Pan (Ant Group, China)

agents reinforcement-learning safety

arXiv Machine Learning

Aug 14

RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning

arXiv:2608. 12146v1 Announce Type: cross Abstract: Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks.

By Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan

reinforcement-learning efficiency

arXiv Machine Learning

Oct 19

RoutePack: Replay-Guided Expert Placement and Rollout Trajectory Scheduling for MoE Reinforcement Learning

RoutePack is a replay‑guided planner that coordinates expert placement and rollout‑trajectory scheduling for Mixture‑of‑Experts reinforcement learning models. By incrementally adjusting expert placement while preserving residency and then rescheduling trajectories into token‑capped micro‑batches, it jointly balances dense‑side work and layer‑wise expert peaks under fixed capacity constraints. Evaluations on 7.9B and 124B hybrid‑attention MoE models across NVIDIA H20‑3e configurations show throughput gains of 4.44‑11.85% over a DP‑local first‑fit‑decreasing baseline and 1.33‑5.28% over expert rerouting alone.

By Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan

reinforcement-learning efficiency

arXiv AI

Jun 3

Libra: Efficient Resource Management for Agentic RL Post-Training

arXiv:2606. 03077v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become a standard post-training paradigm for large language models (LLMs), extending beyond preference alignment to complex reasoning and multi-turn agentic behaviors.

By Kaiwen Chen, Xin Tan, Jingzong Li, Hong Xu

llms agents reinforcement-learning efficiency safety

arXiv Machine Learning

Jul 17

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

arXiv:2607. 14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment.

By Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao, Jun Gao, Harry Zhang, Irvine Lu, Nolan Ho, Lucian Li, Andrew Lei, Cleon Cheng, Steven Chiang, Yihang Zeng, Di Zhang, Rio Yang, Kaijie Chen, Andrew Chen, Pony Ma, Weizhong Zhang, Cheng Jin

agents reinforcement-learning

arXiv Computation and Language

Sep 14

Expert-Space Exploration in MoE Reinforcement Learning

arXiv:2609.13058v1 Announce Type: new Abstract: Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models hav...

By Hongyi He, Zhenghao Lin, Xiao Liu, Peng Cheng, Yan Lu, Yeyun Gong

llms reinforcement-learning

arXiv Machine Learning

Oct 16

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

ReFlow is a system‑algorithm co‑designed framework for efficient reinforcement learning (RL) post‑training of large language models (LLMs). It introduces replay‑driven decoupling, allowing recent trajectories to be reused multiple times to amortize rollout costs while limiting policy staleness. The architecture features a dual replay cycle, an elastic resource pool, and an adaptive scheduler, achieving up to 2.06× higher training throughput compared to existing synchronous and asynchronous RL systems without sacrificing model performance.

By Yukun Tian, Junwei Su, Haofeng Xu, Lansong Diao, Zhengping Qian, Weibo Zhao, Wenchao Zhou, Ruiting Zhou, Chuan Wu

llms agents reinforcement-learning efficiency benchmarks

arXiv Computation and Language

Oct 9

MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

MiMo‑V2.6 is an omni‑modal reinforcement learning framework that scales compute to advance large foundation models toward self‑improvement. It trains on a broad multimodal corpus before RL, then scales RL along three axes: larger batches and throughput, diverse environments across code, general, visual, and cyber domains, and enhanced grader compute for better reward signals. The system incorporates infrastructure such as a frozen MoE router, multi‑layer defense against reward hacking, unified trajectory representation, high‑concurrency rollout, and training‑inference consistency, and it is open‑source to support reproducibility and further research.

By Core Team, Zongming Qiao, Ziyue Hua, Zirui Ou, Zihao Yue, Zihan Jiang, Zhuo Huang, Zhiyang Chen, Zhixian Zheng, Zhipeng Xu, Zhengrui Ma, Yuyang Hu, Yuhang Dong, Yuechen Zhang, Yudong Wang, Yuanxin Liu, Yixin Yang, Yishuo Cai, Yikai Zhao, Yihan Yan, Yifan Zhang, Yifan Song, Xiyu Wei, Xing Zhang, Xin Zhang, Xiaoqian Liu, Xiaodong Ji, Xiangwei Deng, Xueyu Guo, Wenhan Ma, Weimin Xiong, Weikun Wang, Weiji Zhuang, Shuo Liu, Shuhuai Ren, Shuhao Gu, Shimao Chen, Shijie Cao, Shihua Yu, Shicheng Li, Shengjie Zhou, Shaolei Zhang, Rang Li, Qiying Wang, Qingkai Fang, Qianli Chen, Minzheng Wang, Liwen Wang, Linli Yao, Linghao Zhang, Liangyu Cheng, Liang Zhao, Lei Li, Jinhao Dong, Jinyu Xiang, Jianyu Wei, Jiangshan Duo, Huaqiu Liu, Huanjie Fan, Hongyi Guan, Hongshen Xu, Hao Tian, Hanyu Li, Hailin Zhang, Gang Wang, Fuli Luo, Feng Wei, Dong Zhang, Dawei Zhu, Chiheng Lou, Chenhong He, Chenhao He, Chenghua Liu, Bowen Ye, Bowen Shen, Boshen Xu, Bo Yang, Bingquan Xia, Bangjun Xiao, Baixuan Xu, Zhouxiang Mao, Zhiyang Zhang, Zhixiang Xu, Zhenru Lin, Zhengju Tang, Zhaojun Huang, Yuzhe Weng, Yuxing Xiang, Yuxiao Li, Yuheng Yang, Yuhang Wang, Yuchen Liu, Yuanyuan Tian, Yuanliang Dong, Yu Cheng, Yongzhe He, Yongshun Liang, Yong Wang, Yiyan Wang, Yitian Gong, Yijie Zhang, Yanshu Xin, Xun Zhang, Xingjian Zhao, Wenyu Yang, Wenshan Huang, Wenhao Li, Tingwei Huang, Tianyu Yu, Tianyang Lu, Taoyu Yang, Sinan Du, Shutong Tian, Shulin Du, Shengfan Wang, Shanchuan Fang, Qihao Zhang, Qibin Yang, Qian Yu, Qian Tu, Pengrong Xie, Peipei Wang, Peidian Li, Minkun Guo, Mingchen Shao, Luohan Gao, Lijie Wang, Liang Shi, Kaiqi Chen, Kaiming Liu, Kaifei Wang, Kai Yang, Jinlong Xue, Jiechen Zhang, Jiaxuan Liu, Hongxu An, Hao Peng, Hanglong L\"u, Guonan Wang, Feiyu Yang, Fanyu Cao, Fangyue Liu, Fan Cui, Cong Wang, Chun Chen, Chenxu Bai, Chengxuan Zhu, Chenghua Wang, Boyi Zeng

agents reinforcement-learning efficiency multimodal

arXiv Machine Learning

Aug 12

Scheduling Mixed RL Rollouts Beyond Prefix Locality

arXiv:2608. 11152v1 Announce Type: cross Abstract: Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms.

By Zetao Hong, Song Yuan, Yuanhao Ding, Yibo Zhu, Daxin Jiang, Zhibin Wang, Chen Tian

llms agents reinforcement-learning efficiency

arXiv AI

Sep 23

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

arXiv:2605.01457v4 Announce Type: replace Abstract: How can generative offline multi-agent reinforcement learning achieve both fast joint trajectory generation and effective cooperation? Multi-agent...

By Guowei Zou, Haitao Wang, Beiwen Zhang, Boning Zhang, Hejun Wu

agents diffusion reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 17

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

arXiv:2607. 14171v1 Announce Type: new Abstract: Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes.

By Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 21

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

arXiv:2607. 17299v1 Announce Type: cross Abstract: Long-horizon rollout generation has become the dominant systems bottleneck in agentic reinforcement learning (RL).

By Ryan Xu, Atlas Zhao, David Bao, Frank Du

agents reinforcement-learning efficiency

Related stories