arXiv AI By Daize Dong, Junlin Chen, Haolong Jia, Jiawei Wu, Huanwei Di, Jiang Liu, Jialian Wu, Zhengzhong Liu, Zicheng Liu, Emad Barsoum, Dimitris N. Metaxas, Hongyi Wang

PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Read the original on arXiv AI →

arXiv:2606. 00395v1 Announce Type: cross Abstract: Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv Machine Learning
5d ago

RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning

arXiv:2608. 12146v1 Announce Type: cross Abstract: Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks.

By Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan