arXiv:2606. 08814v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) scales model capacity efficiently by selectively routing inputs to a specialized subset of experts.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
arXiv:2604. 00421v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments.
The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
The paper introduces L2R, a routing framework for Mixture-of-Experts models that reshapes the routing space into a shared low‑rank latent space and employs Saturated Inner‑Product Scoring to control Lipschitz behavior, resulting in smoother and more stable routing geometry. It also adds a parameter‑efficient multi‑anchor routing mechanism to increase expert expressiveness. Experiments on an OLMoE‑based language model and a ViT‑based ImageNet setting demonstrate improved overall performance and better routing geometry and expert discrimination.
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
The paper introduces GRIP, an algorithm‑agnostic framework for machine unlearning in Mixture‑of‑Experts large language models. GRIP enforces hard geometric constraints on router updates, projecting gradient changes into the null space of the retain set’s routing matrix to prevent routing manipulation. Two variants—training‑time stochastic projection and post‑training analytical correction—show significant improvements in routing stability, retain accuracy, and resistance to white‑box adversarial recovery across two MoE models.
arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.