arXiv AI
Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 4
Towards a Statistical Understanding of Mixture-of-Experts
The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
arXiv AI
Sep 18
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
The paper introduces L2R, a routing framework for Mixture-of-Experts models that reshapes the routing space into a shared low‑rank latent space and employs Saturated Inner‑Product Scoring to control Lipschitz behavior, resulting in smoother and more stable routing geometry. It also adds a parameter‑efficient multi‑anchor routing mechanism to increase expert expressiveness. Experiments on an OLMoE‑based language model and a ViT‑based ImageNet setting demonstrate improved overall performance and better routing geometry and expert discrimination.
arXiv Computation and Language
Aug 28
Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
MetaNet is a support‑set controller that predicts, for each layer of a Mixture‑of‑Experts model, an expert‑retention threshold and a bounded routing bias while keeping the backbone, experts, and router frozen. On DeepSeek‑MoE‑16B‑Chat, MetaNet offers a tunable trade‑off between accuracy and expert activation: a conservative setting activates 3.61 experts on average (40% fewer than a fixed k=6) with comparable MMLU accuracy, whereas an aggressive setting activates only 2.28 experts (62% fewer) with a modest accuracy drop. The MMLU‑trained controller also transfers to C‑Eval, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
arXiv AI
Sep 17
MoRE: Mixture of Reused Experts
MoRE: Mixture of Reused Experts is a hybrid architecture that combines Mixture-of-Experts (MoE) with weight‑sharing techniques. It shares expert pools across adjacent layers while each layer keeps its own router, and introduces lightweight depth embeddings to help shared experts differentiate layer contexts. Experiments on models ranging from 114 M to 1.15 B parameters show MoRE achieves lower perplexity and better downstream performance than standard MoEs and other weight‑sharing models, with only minimal changes to existing MoE implementations.
arXiv Machine Learning
Oct 22
Generalization and Scaling Laws for Mixture-of-Experts Transformers
The paper presents a theoretical framework for understanding generalization and scaling in Mixture-of-Experts (MoE) Transformers. It derives a sup‑norm covering‑number bound that separates active per‑input capacity from routing combinatorics, leading to a generalization bound under a manifold data model. Additionally, a constructive approximation theorem shows how error can be reduced by scaling active capacity or increasing the number of experts, from which neural scaling laws and compute‑optimal tradeoffs are derived.
arXiv Machine Learning
Oct 7
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
MASKerade is a new dense‑to‑MoE training method that learns experts as sparse subnetworks of a frozen pretrained feed‑forward network (FFN) using binary masks. A token‑level router selects which masked FFNs to execute, and both the router and mask scores are jointly optimized while the underlying FFN weights remain unchanged. In experiments on five vision‑language benchmarks with Qwen and Gemma backbones, a configuration of four 2:4 experts with top‑2 routing achieved the best performance among compared baselines, demonstrating that mask learning over frozen weights is a practical alternative for constructing token‑routed MoE experts.