arXiv Machine Learning By Hao Li, Yasuyuki Tahara, Yuichi Sei

Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

Read the original on arXiv Machine Learning →

arXiv:2609. 17940v1 Announce Type: new Abstract: Sparse mixture-of-experts models route each token through a sequence of expert selections.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Sep 3

Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

The paper investigates how routing decisions in sparse mixture-of-experts (MoE) models evolve across layers. By aligning router control subspaces with generalized orthogonal Procrustes analysis, the authors find that a single linear transition can predict routing states across depth with substantial accuracy, revealing a shared geometric structure. They further demonstrate that these canonical states preserve expert selection better than generic hidden representations and improve next‑step routing predictions, reducing negative log‑likelihood by up to 15.7% on OLMoE and 6.2% on Phi.

By Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov, Artem Gorokhov
arXiv Machine Learning
Aug 27

Output Dilution: Redundant but Fragile Representations in MoE Models

The paper investigates how Mixture-of-Experts (MoE) models encode moral content compared to dense models. While linear probes can recover moral valence from almost all expert-layer combinations with high accuracy, these representations are far more fragile to activation noise, showing a 4.2‑fold drop in robustness. The authors attribute this fragility to output dilution: the MoE block averages across active experts, reducing the feedforward signal by nearly two orders of magnitude, which makes moral information vulnerable to perturbation even though routing remains stable.

By Orion Reblitz-Richardson
arXiv AI
Oct 21

The Router Already Knows

The paper investigates whether a sparse dictionary can reuse the routing decisions already made by a mixture‑of‑experts (MoE) model. Using the NR‑SAE approach, each routed expert is given a small dictionary that is opened only when the frozen router selects that expert. Experiments on Gemma 4 26B‑A4B show that the expert‑aligned autoencoder explains 51.8 % of the variance in router‑weighted expert outputs, outperforming several control variants and suggesting that the router already encodes useful information that a dictionary would otherwise need to learn.

By Andrew Bistras, Marty Johnson
arXiv AI
Oct 31

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.

By Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli