arXiv Machine Learning

OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit

OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.

arXiv AI
Nov 2

Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

The paper introduces a structure‑preserving approach to prune experts in Mixture‑of‑Experts large language models, addressing memory overhead while maintaining performance. It observes that expert outputs exhibit structured redundancy, with stable rank and entropy effective rank far below the nominal number of experts. Two pruning methods—Projection Energy Expert Pruning (PEEP) and Distributional Coverage Expert Pruning (DCEP)—use router‑derived weights to preserve global structural relationships and local importance, achieving superior results on DeepSeek‑V2‑Lite and Qwen3‑30B‑A3B at 75% and 50% expert retention.

By Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li
arXiv Machine Learning
Oct 3

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

MoRA is a framework for pruning Mixture-of-Experts (MoE) models by learning a router bias for each expert and optimizing it with a language‑modeling loss and a routing‑diversity regularizer. The learned biases sharpen routing distributions to identify critical experts and encourage diverse routing preferences. After pruning, MoRA uses an expert approximation mechanism that approximates the outputs of pruned experts with affine transformations of remaining experts, further improving performance.

By Yushuai Sun, Zikun Zhou, Lin Gao, Jun Yu, Wenjie Pei
arXiv Machine Learning
Aug 27

ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.

By Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan