arXiv AI October 16, 2026 By Liangyao Jin, Zhixin Wang, Liming Liu, Mingjun Zhang, Zhengzhou Zhu, Fangcheng Fu, Jinlong Hou, Yuan Cheng, Xiaohe Hu, Ping Zhang, Siyuan Feng

Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

Read the original on arXiv AI →

The paper introduces Router Drift Control (RDC), a method for mitigating off‑policy degradation in Mixture‑of‑Experts reinforcement learning. RDC uses the disagreement between rollout and training routers to detect stale routing decisions and selectively rejects high‑drift responses, thereby avoiding the gradient bias introduced by Rollout Routing Replay (R3). Experiments show that RDC improves validation accuracy, especially in asynchronous training settings, outperforming both R3 and Truncated Importance Sampling (TIS).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 2

PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

arXiv:2606. 00395v1 Announce Type: cross Abstract: Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale.

By Daize Dong, Junlin Chen, Haolong Jia, Jiawei Wu, Huanwei Di, Jiang Liu, Jialian Wu, Zhengzhong Liu, Zicheng Liu, Emad Barsoum, Dimitris N. Metaxas, Hongyi Wang

llms reinforcement-learning benchmarks

arXiv Computation and Language

Sep 14

Expert-Space Exploration in MoE Reinforcement Learning

arXiv:2609.13058v1 Announce Type: new Abstract: Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models hav...

By Hongyi He, Zhenghao Lin, Xiao Liu, Peng Cheng, Yan Lu, Yeyun Gong

llms reinforcement-learning

arXiv Machine Learning

Oct 15

MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning

arXiv:2610.20238v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) excel on graph-structured data but suffer catastrophic forgetting when classes arrive sequentially. In class-incremental g...

By Junlin Chen, Wenhao Wang, Junchao Xu, Qingsong Yao, Fei Xue, Li Sun, Xiao Wang, Ruijie Wang

fine-tuning benchmarks safety

arXiv Machine Learning

Oct 7

Structuring MoE Expert Selection for Agentic Reinforcement Learning

arXiv:2610.07332v1 Announce Type: new Abstract: Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structur...

By Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 22

Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

arXiv:2607. 18722v1 Announce Type: new Abstract: Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing.

By Junyao Yang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Ruhan Wang, Xiangxin Zhou, Kishan Panaganti, Haitao Mi, Leowei Liang

reinforcement-learning efficiency

arXiv Machine Learning

Oct 8

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

The paper introduces COPC, a Coupled Off-Policy Correction method for asynchronous reinforcement learning of large language models. COPC coordinates policy-side and advantage-side corrections by combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals, addressing both policy mismatch and advantage staleness. Experiments show COPC outperforms existing asynchronous baselines on tool-integrated mathematical reasoning and search tasks, while maintaining training stability and minimal overhead.

By Zicheng Hu, Zhijian Zhou, Xuan Zhang, Yuchen Liu, Cheng Chen, Yuan Li, Qi Gu, Yan Feng, Hongyan Hao, Chao Qu

llms reinforcement-learning safety

One email a morning, machine-written

Related stories

PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Expert-Space Exploration in MoE Reinforcement Learning

MAEM: Mitigating Recency Bias via a Replay-Free, Capacity-Controlled Dynamic Mixture of Experts for Class-Incremental Graph Learning

Structuring MoE Expert Selection for Agentic Reinforcement Learning

Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning