arXiv Machine Learning

Generalization and Scaling Laws for Mixture-of-Experts Transformers

The paper presents a theoretical framework for understanding generalization and scaling in Mixture-of-Experts (MoE) Transformers. It derives a sup‑norm covering‑number bound that separates active per‑input capacity from routing combinatorics, leading to a generalization bound under a manifold data model. Additionally, a constructive approximation theorem shows how error can be reduced by scaling active capacity or increasing the number of experts, from which neural scaling laws and compute‑optimal tradeoffs are derived.

arXiv Machine Learning

Sep 4

Towards a Statistical Understanding of Mixture-of-Experts

The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.

By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang

arXiv AI

Jun 10

Mixtures of Neural Operators Reduce Active Complexity in Operator Learning

arXiv:2404. 09101v3 Announce Type: replace-cross Abstract: Operator-learning systems are not governed solely by total parameter count; for one query, the relevant bottleneck can be the model that must be loaded and evaluated.

By Anastasis Kratsios, Takashi Furuya, Jose Antonio Lara Benitez, Matti Lassas, Maarten de Hoop

arXiv AI

Oct 15

Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation

arXiv:2610.14644v1 Announce Type: new Abstract: Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing...

By Yaoxiang Wang, Kai Zhang, Ruobing Xie, Lanrui Wang, Guoliang Zhao, Junhui Li, Jinsong Su

arXiv AI

Oct 15

Optimal Transport Aggregation for Distributed Mixture-of-Experts

The paper addresses how to combine independently fitted local Mixture-of-Experts (MoE) models from distributed datasets into a single global MoE. It proposes reducing the resulting large mixture to a K‑expert model by minimizing an expected transportation divergence that depends on covariates, using a majorization–minimization algorithm for experts and a separate fitting for the gating network. The method guarantees consistency of the aggregated estimator, requires only one one‑way communication round, and achieves accuracy close to centralized estimation while cutting computation time.

By Fa\"icel Chamroukhi, Nhat Thien Pham

arXiv AI

Oct 7

Distributionally Robust Mixture-of-Experts Training

arXiv:2610.07207v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability probl...

By Xin Teng, Muxiao Li, Hongyi Wen

llms efficiency

arXiv AI

Jun 3

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

arXiv:2512. 13996v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs.

By Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohi Dibua, Yifan Gong, Yan Kang, Dimitris N. Metaxas

llms diffusion efficiency

arXiv AI

Jun 11

Unifying Learning Dynamics and Generalization in Transformers Scaling Law

arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.

By Chiwun Yang

llms

arXiv AI

Aug 12

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

arXiv:2608. 10605v1 Announce Type: cross Abstract: In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages.

By Soumajyoti Sarkar, Yuxin Tang, Sheng Zha

llms efficiency

Hugging Face Trending Papers

Aug 11

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency.

llms efficiency

arXiv AI

Jun 2

DOT-MoE: Differentiable Optimal Transport for MoEfication

arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.

By Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta

llms efficiency benchmarks

arXiv Computation and Language

Sep 10

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

arXiv:2609.09241v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large f...

By Dohyeon Kim, Bedionita Soro, Sung Ju Hwang

llms benchmarks safety

arXiv AI

Jun 2

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.

By Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng

benchmarks

Related stories