The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
The paper presents a theoretical framework for understanding generalization and scaling in Mixture-of-Experts (MoE) Transformers. It derives a sup‑norm covering‑number bound that separates active per‑input capacity from routing combinatorics, leading to a generalization bound under a manifold data model. Additionally, a constructive approximation theorem shows how error can be reduced by scaling active capacity or increasing the number of experts, from which neural scaling laws and compute‑optimal tradeoffs are derived.
By Mansour Zoubeirou a Mayaki
arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.
By Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta
arXiv:1907.06994v2 Announce Type: replace-cross
Abstract: Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predi...
By Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van, Bao Tuyen Huynh
arXiv:2609.09241v1 Announce Type: cross
Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large f...
By Dohyeon Kim, Bedionita Soro, Sung Ju Hwang
The paper introduces Elastic Expert Routing, a method that replaces the fixed top‑k expert selection in Sparse Mixture‑of‑Experts models with a stochastic sampling of the expert budget around k. This approach smooths the hard boundary between fully supervised and zero‑feedback experts, maintaining the same expected compute cost while improving training stability. Experiments show that elastic routing boosts downstream performance on supervised fine‑tuning and from‑scratch pretraining, achieving higher macro‑averages on OLMoE‑1B‑7B, Qwen3‑30B‑A3B, and outperforming static top‑k baselines by an average of 1.6 points.
By Yunkai Chai, Tong Zhu, Xiaoye Qu, Xuyang Hu, Guanjie Chen, Qipeng Guo, Yu Cheng
arXiv:2511. 08972v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) models are scalable and computationally efficient, enabling large increases in model capacity with limited inference overhead.
By Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan Minh Nguyen, Toan Tran
arXiv:2608. 16101v1 Announce Type: cross Abstract: Coresets distill large datasets into small, representative subsets for efficient downstream learning.
By Haoyun Yin, Chuanhui Liu, Xiao Wang
arXiv:2608. 08627v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank.
By Lei Xin, Bin Gu, Peize Li, Zitong Wang, Jianbo Zhao, Changjiang Jiang, Yanyue Xie, Chao Huang, Xuyang Zhao, Zunhai Su, Fanhu Zeng, Zhenglun Kong
The paper introduces DS-MoE, a framework that selects a compact subset of experts in Mixture-of-Experts models using difference-of-submodular (DS) optimization. It analyzes the loss degradation via a second-order Taylor expansion to uncover redundancy and synergy among experts, and formulates the selection objective to decouple these effects. A majorization-minimization algorithm with monotonicity guarantees efficiently identifies the optimal subset, and experiments show DS-MoE outperforms state‑of‑the‑art baselines.
By Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu, Zihan Fang, Zhe Chen, Wei Ni, Jun Luo, Symeon Chatzinotas
arXiv:2610.07804v1 Announce Type: new
Abstract: Prior Fitted Networks (PFNs) such as TabPFN now rival established statistical procedures across prediction and estimation tasks. A natural explanation...
By Martin Eppert, Krishna Balasubramanian, Subhro Ghosh, Jason Klusowski, Yan Shuo Tan
arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.
By Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng