← Back to all news
arXiv Machine Learning By Mansour Zoubeirou a Mayaki

Generalization and Scaling Laws for Mixture-of-Experts Transformers

Read the original on arXiv Machine Learning →

The paper presents a theoretical framework for understanding generalization and scaling in Mixture-of-Experts (MoE) Transformers. It derives a sup‑norm covering‑number bound that separates active per‑input capacity from routing combinatorics, leading to a generalization bound under a manifold data model. Additionally, a constructive approximation theorem shows how error can be reduced by scaling active capacity or increasing the number of experts, from which neural scaling laws and compute‑optimal tradeoffs are derived.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 4

Towards a Statistical Understanding of Mixture-of-Experts

The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.

By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
More like this →
arXiv AI
Oct 15

Optimal Transport Aggregation for Distributed Mixture-of-Experts

The paper addresses how to combine independently fitted local Mixture-of-Experts (MoE) models from distributed datasets into a single global MoE. It proposes reducing the resulting large mixture to a K‑expert model by minimizing an expected transportation divergence that depends on covariates, using a majorization–minimization algorithm for experts and a separate fitting for the gating network. The method guarantees consistency of the aggregated estimator, requires only one one‑way communication round, and achieves accuracy close to centralized estimation while cutting computation time.

By Fa\"icel Chamroukhi, Nhat Thien Pham
More like this →
arXiv AI
Jun 3

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

arXiv:2512. 13996v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs.

By Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohi Dibua, Yifan Gong, Yan Kang, Dimitris N. Metaxas
llmsdiffusionefficiency
More like this →