arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.
By Yushu Zhao, Zheng Wang, Minjia Zhang
arXiv:2610.27123v1 Announce Type: cross
Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert p...
By Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao, Zhipeng Wang, Shayan Mohajer Hamidi, Yuanda Xu, Zhengze Zhou, Wenhui Zhu, Rahul Mazumder
arXiv:2606. 18304v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead.
By Yifu Ding, Jiacheng Wang, Ge Yang, Yongcheng Jing, Jinyang Guo, Xianglong Liu, Dacheng Tao
arXiv:2610.32009v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse activation, but storing their expert parameters remains a major memory bottleneck. We...
By Carson Cary, Yukai Jin, Yihao Huang, Zhankai Ye, Fangyi Li, Ting Hua, Xin Liu, Shangqian Gao
The paper introduces a structure‑preserving approach to prune experts in Mixture‑of‑Experts large language models, addressing memory overhead while maintaining performance. It observes that expert outputs exhibit structured redundancy, with stable rank and entropy effective rank far below the nominal number of experts. Two pruning methods—Projection Energy Expert Pruning (PEEP) and Distributional Coverage Expert Pruning (DCEP)—use router‑derived weights to preserve global structural relationships and local importance, achieving superior results on DeepSeek‑V2‑Lite and Qwen3‑30B‑A3B at 75% and 50% expert retention.
By Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li
arXiv:2606. 09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts.
By Jiangyang He, Shaolin Zhu, Deyi Xiong