arXiv Machine Learning

Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

arXiv:2609. 17940v1 Announce Type: new Abstract: Sparse mixture-of-experts models route each token through a sequence of expert selections.

arXiv AI
Sep 3

Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

The paper investigates how routing decisions in sparse mixture-of-experts (MoE) models evolve across layers. By aligning router control subspaces with generalized orthogonal Procrustes analysis, the authors find that a single linear transition can predict routing states across depth with substantial accuracy, revealing a shared geometric structure. They further demonstrate that these canonical states preserve expert selection better than generic hidden representations and improve next‑step routing predictions, reducing negative log‑likelihood by up to 15.7% on OLMoE and 6.2% on Phi.

By Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov, Artem Gorokhov
arXiv Machine Learning
Aug 27

Output Dilution: Redundant but Fragile Representations in MoE Models

The paper investigates how Mixture-of-Experts (MoE) models encode moral content compared to dense models. While linear probes can recover moral valence from almost all expert-layer combinations with high accuracy, these representations are far more fragile to activation noise, showing a 4.2‑fold drop in robustness. The authors attribute this fragility to output dilution: the MoE block averages across active experts, reducing the feedforward signal by nearly two orders of magnitude, which makes moral information vulnerable to perturbation even though routing remains stable.

By Orion Reblitz-Richardson
arXiv AI
Oct 21

The Router Already Knows

The paper investigates whether a sparse dictionary can reuse the routing decisions already made by a mixture‑of‑experts (MoE) model. Using the NR‑SAE approach, each routed expert is given a small dictionary that is opened only when the frozen router selects that expert. Experiments on Gemma 4 26B‑A4B show that the expert‑aligned autoencoder explains 51.8 % of the variance in router‑weighted expert outputs, outperforming several control variants and suggesting that the router already encodes useful information that a dictionary would otherwise need to learn.

By Andrew Bistras, Marty Johnson
arXiv AI
Oct 31

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.

By Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
arXiv Machine Learning
Sep 4

Towards a Statistical Understanding of Mixture-of-Experts

The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.

By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
arXiv Machine Learning
Sep 22

Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation

The paper studies how the design of Mixture-of-Experts (MoE) routers affects inference speed when combined with Speculative Decoding (SD). It shows that routers promoting high expert coactivation reduce memory transfer costs and improve runtime. By integrating a global load‑balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism, the authors achieve a 21% throughput gain over baseline MoEs while preserving accuracy.

By Kumari Nishu, Han-Byul Kim, Santosh Chilkunda, Maxwell Horton, Arnav Kundu, Mohammad Samragh, Lauren Hannah, Mohammad Sekhavat, Nikhil Bhendawade, Manuel Ciosici, Iman Mirzadeh, Keivan Alizadeh Vahid, David Harrison, Irina Belousova, Mehrdad Farajtabar, Minsik Cho
arXiv Computation and Language
Nov 4

Multi-View Latent MoE: Letting Every Router Shape Expert Computation

The paper introduces Multi-View Latent MoE, a sparse mixture‑of‑experts architecture that uses multiple independent routers to generate richer selection signals and to modulate expert computation. By aggregating sigmoid scores from these routers, the model selects experts that are strongly favored by one or several routers and mixes their relative scores into each expert’s input, thereby reusing routing preferences for both selection and computation. Experiments on a 24‑layer 10B‑A1B model trained on 708 B tokens show a 1.18‑point improvement in nine‑task downstream mean over the baseline, with the Multi‑Router submodule alone providing a 0.6‑point gain and significantly better alignment between router scores and expert utility. whyItMatters":"The approach demonstrates that leveraging multiple routers can both improve expert selection quality and maintain computational efficiency, leading to measurable downstream performance gains on large‑scale language models."

By Zelin Zhao, Junyuan Shang, Tianchi Ma, Shuohuan Wang, Dianhai Yu
arXiv Machine Learning
Sep 30

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.

By Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu, Bei Liu, Sirui Han, Yike Guo