OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.
By Dezhi Li, Lujun Li, Qiyuan Zhu, Hao Gu, Bei Liu, Sirui Han, Yike Guo
arXiv:2609.38823v1 Announce Type: new
Abstract: Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly becau...
By Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang, Yaoxin Yang, Jiayuan Fan, Tao Chen
ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.
By Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan
arXiv:2606. 05538v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges.
By Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, Heather Miller
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath
arXiv:2608. 14385v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems.
By Zewen Jin, Shen Fu, Zeping Duan, Shannon Wang, Weihao Wu, Chengjie Tang, Congkun Ai, Ping Gong, Zijian Dai, Youhui Bai, Cheng Li
PCoMoE introduces a path‑compositional execution framework that moves Mixture‑of‑Experts inference from coarse‑grained expert selection to fine‑grained path composition. It uses a path‑level formulation of expert computation, a compatibility‑aware layer‑wise pruning strategy to eliminate low‑value path combinations, and a hardware‑friendly execution engine that reuses sub‑expert structures with bounded overhead. Experiments show up to a 1.31× speedup and a 10% accuracy improvement over existing MoE inference methods.
By Ziyan Gan, Fangxin Liu, Chenyang Guan, Junjie Wang, Ning Yang, Haomin Li, Xiang Li, Siran Yang, Jiamang Wang, Lin Qu, Zongwu Wang, Li Jiang, Haibing Guan
Residual sparsification via output importance (PARSER) is a new compression technique for mixture-of-experts large language models that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. By introducing output importance, PARSER measures each residual’s contribution to the final expert output and compresses accordingly. Experiments show that PARSER reduces the accuracy gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek while achieving the same peak memory reduction.
By Seungwoo Jung, Dohyeok Kwon, Seungmin Cha, Junseok Lee, Yeonho Yoo, Chuck Yoo, Gyeongsik Yang
Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrain...
Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning.
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You
arXiv:2606. 01062v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge.
By Jiarui Feng, Hanqing Zeng, Karish Grover, Ruizhong Qiu, Yinglong Xia, Qiang Zhang, Qifan Wang, Ren Chen, Dongqi Fu, Jiayi Liu, Zhoukai Zhao, Xiangjun Fan, Benyu Zhang, Yixin Chen