arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.
By Yushu Zhao, Zheng Wang, Minjia Zhang
arXiv:2610.27123v1 Announce Type: cross
Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert p...
By Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao, Zhipeng Wang, Shayan Mohajer Hamidi, Yuanda Xu, Zhengze Zhou, Wenhui Zhu, Rahul Mazumder
arXiv:2606. 18304v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead.
By Yifu Ding, Jiacheng Wang, Ge Yang, Yongcheng Jing, Jinyang Guo, Xianglong Liu, Dacheng Tao
arXiv:2610.32009v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We...
By Carson Cary, Yukai Jin, Yihao Huang, Zhankai Ye, Fangyi Li, Ting Hua, Xin Liu, Shangqian Gao
The paper introduces a structure‑preserving approach to prune experts in Mixture‑of‑Experts large language models, addressing memory overhead while maintaining performance. It observes that expert outputs exhibit structured redundancy, with stable rank and entropy effective rank far below the nominal number of experts. Two pruning methods—Projection Energy Expert Pruning (PEEP) and Distributional Coverage Expert Pruning (DCEP)—use router‑derived weights to preserve global structural relationships and local importance, achieving superior results on DeepSeek‑V2‑Lite and Qwen3‑30B‑A3B at 75% and 50% expert retention.
By Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li
arXiv:2606. 09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts.
By Jiangyang He, Shaolin Zhu, Deyi Xiong
MoRA is a framework for pruning Mixture-of-Experts (MoE) models by learning a router bias for each expert and optimizing it with a language‑modeling loss and a routing‑diversity regularizer. The learned biases sharpen routing distributions to identify critical experts and encourage diverse routing preferences. After pruning, MoRA uses an expert approximation mechanism that approximates the outputs of pruned experts with affine transformations of remaining experts, further improving performance.
By Yushuai Sun, Zikun Zhou, Lin Gao, Jun Yu, Wenjie Pei
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath
arXiv:2606. 09886v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) large language models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall: the full expert pool must remain resident to support token-dependent routing.
By Yuhao Zhang
ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.
By Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan
arXiv:2609.38823v1 Announce Type: new
Abstract: Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly becau...
By Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang, Yaoxin Yang, Jiayuan Fan, Tao Chen