arXiv Machine Learning

FlexMARL: Graph-Driven Rollout-Training Co-Design for LLM-Based Multi-Agent Reinforcement Learning

FlexMARL is a graph-driven rollout‑training co‑designed framework that addresses the infrastructure gap in large‑scale LLM‑based multi‑agent reinforcement learning. It introduces a multi‑agent execution graph (MEG) and a MEG Store for decentralized storage, a Rollout Engine that enables dependency‑parallel sampling with hierarchical load balancing, a Joint Orchestrator that uses an agent‑level asynchronous pipeline to reduce synchronization barriers, and a Training Engine that allocates resources on demand and swaps training states via location‑agnostic APIs. Experiments on a production cluster show up to 7.3× speedup and 5.6× better hardware utilization compared to existing frameworks.

arXiv AI
Oct 9

AgentFly: Scaling Agentic Reinforcement Learning with Unified Resource System

AgentFly is an agentic reinforcement learning framework that introduces a unified resource layer to manage heterogeneous environments—such as sandboxes, model services, and external APIs—through a single scheduling engine. The framework is organized into four layers: an agent layer that abstracts agents, tools, and rewards; a rollout layer that composes agent loops and computes rewards; a context layer that organizes rollouts and injects contextual information; and a low‑level resource layer that handles resource management. AgentFly offers prebuilt tools and environments, demonstrates successful agent training across multiple tasks and models, and provides the first controlled cross‑framework throughput comparison against other agentic RL frameworks.

By Renxi Wang, Rifo Ahmad Genadi, Bilal El Bouardi, Yongxin Wang, Fajri Koto, Zhengzhong Liu, Timothy Baldwin, Haonan Li
arXiv Machine Learning
Oct 16

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

ReFlow is a system‑algorithm co‑designed framework for efficient reinforcement learning (RL) post‑training of large language models (LLMs). It introduces replay‑driven decoupling, allowing recent trajectories to be reused multiple times to amortize rollout costs while limiting policy staleness. The architecture features a dual replay cycle, an elastic resource pool, and an adaptive scheduler, achieving up to 2.06× higher training throughput compared to existing synchronous and asynchronous RL systems without sacrificing model performance.

By Yukun Tian, Junwei Su, Haofeng Xu, Lansong Diao, Zhengping Qian, Weibo Zhao, Wenchao Zhou, Ruiting Zhou, Chuan Wu
arXiv AI
Jun 16

RollArt: Disaggregated Multi-Task Agentic RL Training at Scale

arXiv:2512. 22560v2 Announce Type: replace-cross Abstract: Agentic Reinforcement Learning (RL) trains LLMs through multi-turn interactions with environments, producing workloads that mix compute-bound prefill, bandwidth-bound decoding, CPU-heavy environment execution, and bursty reward evaluation.

By Wei Gao, Yuheng Zhao, Tianyuan Wu, Shaopan Xiong, Weixun Wang, Dakai An, Lunxi Cao, Dilxat Muhtar, Zichen Liu, Haizhou Zhao, Ju Huang, Siran Yang, Yongbin Li, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, Wei Wang
arXiv Machine Learning
Oct 19

LoongSage: A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes

LoongSage is a lightweight, fully asynchronous reinforcement learning framework designed for large language models that perform agentic tasks. It integrates with Megatron for distributed training and SGLang for high‑throughput inference, centralizing orchestration in a single controller and using Ray actors for GPU components. The framework introduces AgentFlow for unified execution of white‑box and black‑box agents, sliding‑window strategies to limit trajectory staleness, TransferMesh for efficient weight synchronization, and a memory‑efficient Teacher Manager that supports multi‑teacher on‑policy distillation. Experiments across four training runs show steady reward improvement, controlled train‑inference gaps, and sustained hardware utilization by eliminating synchronous idle periods.

By LoongSage Team, Bowei Song, Caiwei Zhao, Chuwen Liu, Dechuang Gu, Dong Fang, Feiyu Xie, Huaiyu Li, Hui Zhang, Jian Tian, Jian Wang, Jie Zhao, Liguang Li, Maoyun Wang, Meiyan Yang, Minghan Zhu, Pai Liu, Pandeng Yao, Renbao Tian, Ru Yang, Shuai Jia, Shuai Tan, Shuailong Li, Shuliang Li, Teng Jin, Tianze Yang, Wenbo Cui, Wenjing Bai, Wenkang Wu, Xiaowei Jiang, Xiaoyi Dong, Yanpeng Wang, Yanwen Wang, Yaohui Ma, Yawen Wang, Yihang Zhang, Yingjie Yan, Zhanbiao Lian, Zhenyu Lu, Zhida Hu, Zhihui Hao, Zhihui Hu, Zhiwen Xiao, Zhiyi Gao, Zhongwei Qi, Zhuangzhuang Wu, Ziqian Liu, Ziyue Zheng