arXiv Machine Learning By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang

Towards a Statistical Understanding of Mixture-of-Experts

Read the original on arXiv Machine Learning →

The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 22

GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

GaP-MoE introduces a method for explicit expert specialization in Sparse Mixture-of-Experts models by manipulating expert output gradients instead of adding auxiliary loss terms. The technique selects a leader expert from the top‑K routed experts and projects follower experts’ gradients onto the orthogonal complement of the leader’s recent gradients, reducing directional overlap. Experiments on fine‑tuning BERT for GLUE and pretraining GPT‑2 on various corpora show that GaP‑MoE matches or surpasses standard SMoE performance while remaining a drop‑in module.

By Donghyun Jeong, Yoon-Yeong Kim
arXiv Computation and Language
Nov 4

Multi-View Latent MoE: Letting Every Router Shape Expert Computation

The paper introduces Multi-View Latent MoE, a sparse mixture‑of‑experts architecture that uses multiple independent routers to generate richer selection signals and to modulate expert computation. By aggregating sigmoid scores from these routers, the model selects experts that are strongly favored by one or several routers and mixes their relative scores into each expert’s input, thereby reusing routing preferences for both selection and computation. Experiments on a 24‑layer 10B‑A1B model trained on 708 B tokens show a 1.18‑point improvement in nine‑task downstream mean over the baseline, with the Multi‑Router submodule alone providing a 0.6‑point gain and significantly better alignment between router scores and expert utility. whyItMatters":"The approach demonstrates that leveraging multiple routers can both improve expert selection quality and maintain computational efficiency, leading to measurable downstream performance gains on large‑scale language models."

By Zelin Zhao, Junyuan Shang, Tianchi Ma, Shuohuan Wang, Dianhai Yu
arXiv Machine Learning
Oct 7

MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

MASKerade is a new dense‑to‑MoE training method that learns experts as sparse subnetworks of a frozen pretrained feed‑forward network (FFN) using binary masks. A token‑level router selects which masked FFNs to execute, and both the router and mask scores are jointly optimized while the underlying FFN weights remain unchanged. In experiments on five vision‑language benchmarks with Qwen and Gemma backbones, a configuration of four 2:4 experts with top‑2 routing achieved the best performance among compared baselines, demonstrating that mask learning over frozen weights is a practical alternative for constructing token‑routed MoE experts.

By Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu
arXiv Machine Learning
Oct 9

Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

The paper introduces Elastic Expert Routing, a method that replaces the fixed top‑k expert selection in Sparse Mixture‑of‑Experts models with a stochastic sampling of the expert budget around k. This approach smooths the hard boundary between fully supervised and zero‑feedback experts, maintaining the same expected compute cost while improving training stability. Experiments show that elastic routing boosts downstream performance on supervised fine‑tuning and from‑scratch pretraining, achieving higher macro‑averages on OLMoE‑1B‑7B, Qwen3‑30B‑A3B, and outperforming static top‑k baselines by an average of 1.6 points.

By Yunkai Chai, Tong Zhu, Xiaoye Qu, Xuyang Hu, Guanjie Chen, Qipeng Guo, Yu Cheng