arXiv:2610.07207v1 Announce Type: cross
Abstract: Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability probl...
By Xin Teng, Muxiao Li, Hongyi Wen
arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.
By Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng
The paper investigates a dense analogue of Sparse Mixture-of-Experts (MoE) models by using $K$ SwiGLU experts that are all active for every token and combined via a softmax gate, keeping the total feed‑forward network (FFN) width fixed. Validation loss shows a non‑monotonic relationship with $K$: $K=2$ slightly improves performance over the single‑expert baseline, while $K=4$ and $K=6$ degrade it. The study also finds that the gating mechanism remains largely soft and balanced, except for a near one‑hot routing in the first layer of the $K=4$ model, which is functionally important as forcing uniform routing increases loss significantly.
By Vu Quang Hoang, Nghia Hieu Nguyen
The paper investigates how Mixture-of-Experts (MoE) models encode moral content compared to dense models. While linear probes can recover moral valence from almost all expert-layer combinations with high accuracy, these representations are far more fragile to activation noise, showing a 4.2‑fold drop in robustness. The authors attribute this fragility to output dilution: the MoE block averages across active experts, reducing the feedforward signal by nearly two orders of magnitude, which makes moral information vulnerable to perturbation even though routing remains stable.
By Orion Reblitz-Richardson
The paper introduces Elastic Expert Routing, a method that replaces the fixed top‑k expert selection in Sparse Mixture‑of‑Experts models with a stochastic sampling of the expert budget around k. This approach smooths the hard boundary between fully supervised and zero‑feedback experts, maintaining the same expected compute cost while improving training stability. Experiments show that elastic routing boosts downstream performance on supervised fine‑tuning and from‑scratch pretraining, achieving higher macro‑averages on OLMoE‑1B‑7B, Qwen3‑30B‑A3B, and outperforming static top‑k baselines by an average of 1.6 points.
By Yunkai Chai, Tong Zhu, Xiaoye Qu, Xuyang Hu, Guanjie Chen, Qipeng Guo, Yu Cheng
arXiv:2609.16179v1 Announce Type: new
Abstract: Z-loss has been widely applied to the logits of language-model output heads and sparse mixture-of-experts routers. Z-loss constrains the softmax log-no...
By Bum Jun Kim
MASKerade is a new dense‑to‑MoE training method that learns experts as sparse subnetworks of a frozen pretrained feed‑forward network (FFN) using binary masks. A token‑level router selects which masked FFNs to execute, and both the router and mask scores are jointly optimized while the underlying FFN weights remain unchanged. In experiments on five vision‑language benchmarks with Qwen and Gemma backbones, a configuration of four 2:4 experts with top‑2 routing achieved the best performance among compared baselines, demonstrating that mask learning over frozen weights is a practical alternative for constructing token‑routed MoE experts.
By Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu
arXiv:2604.23036v2 Announce Type: replace-cross
Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...
By Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller
GaP-MoE introduces a method for explicit expert specialization in Sparse Mixture-of-Experts models by manipulating expert output gradients instead of adding auxiliary loss terms. The technique selects a leader expert from the top‑K routed experts and projects follower experts’ gradients onto the orthogonal complement of the leader’s recent gradients, reducing directional overlap. Experiments on fine‑tuning BERT for GLUE and pretraining GPT‑2 on various corpora show that GaP‑MoE matches or surpasses standard SMoE performance while remaining a drop‑in module.
By Donghyun Jeong, Yoon-Yeong Kim
The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
Prior Fitted Networks (PFNs) such as TabPFN now rival established statistical procedures across prediction and estimation tasks. A natural explanation is that PFNs have the property of statistical ada...
MetaNet is a support‑set controller that predicts, for each layer of a Mixture‑of‑Experts model, an expert‑retention threshold and a bounded routing bias while keeping the backbone, experts, and router frozen. On DeepSeek‑MoE‑16B‑Chat, MetaNet offers a tunable trade‑off between accuracy and expert activation: a conservative setting activates 3.61 experts on average (40% fewer than a fixed k=6) with comparable MMLU accuracy, whereas an aggressive setting activates only 2.28 experts (62% fewer) with a modest accuracy drop. The MMLU‑trained controller also transfers to C‑Eval, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.
By Rongfeng Wang, Shichao Weng, Zhiqiang Wang, Xinyu Liu, Yang Yi, Peilong Zhou, Hongwei Tang