arXiv Computation and Language
Expert-Space Exploration in MoE Reinforcement Learning
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Aug 14
RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning
arXiv:2608. 12146v1 Announce Type: cross Abstract: Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks.
arXiv AI
Jun 2
PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning
arXiv:2606. 00395v1 Announce Type: cross Abstract: Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale.
arXiv Machine Learning
Aug 27
Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
arXiv Computation and Language
Sep 10
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
arXiv:2604.23036v2 Announce Type: replace-cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...
arXiv Machine Learning
Aug 12
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
arXiv AI
Sep 7
ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs
ACE introduces a training‑free, calibration‑free framework for adaptive expert skipping in Mixture‑of‑Experts LLMs. It combines a Global Spectral Proxy that estimates global transformation capacity with a Router‑Conditioned Refinement that builds expert‑specific direction prototypes, enabling the model to skip low‑contribution experts while always keeping the top‑1 expert. Offline computation of expert statistics leaves only lightweight table lookups during inference, and experiments on three MoE‑based LLMs show ACE outperforms static and dynamic baselines, especially at high skipping ratios.