The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
By Gongli Zhang, Zhulin Liu, C. L. Philip Chen
arXiv:2607. 08780v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices.
By Ali Kayyam
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You
The paper introduces SAME (Stabilized Mixture-of-Experts) to address challenges in Multimodal Continual Instruction Tuning (MCIT) for large language models. SAME mitigates router drift by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions, while preventing expert drift through curvature‑aware scaling that uses historical input covariance without rehearsal. The method also employs adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross‑task interference, and demonstrates state‑of‑the‑art performance on a new long‑task‑sequence benchmark.
By Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou
arXiv:2606. 15453v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.
By Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth, Ke Wang
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.
By Jingyi Xie, Yijun Lin, Yinjiang Xiong, Zhikun Zhang, Sai Li
arXiv:2604. 00421v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments.
By Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli
arXiv:2608. 02989v1 Announce Type: new Abstract: Speculative decoding verifies a tree of draft tokens in one target-model forward pass.
By Shuang Liang (Mark), Hao (Mark), Chen, Zhiwen Mo, Qianzhou Wang, Guoyu Li, Lingxiao Ma, Wayne Luk
arXiv:2608. 12146v1 Announce Type: cross Abstract: Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks.
By Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan
arXiv:2602. 19938v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets.
By Zijie Liu, Jie Peng, Jinhao Duan, Zirui Liu, Kaixiong Zhou, Mingfu Liang, Luke Simon, Xi Liu, Zhaozhuo Xu, Tianlong Chen