arXiv Machine Learning

You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

arXiv AI
Sep 17

Higher-order pruning of experts in mixture-of-experts language models

The paper introduces HOPE, a second‑order pruning method for Mixture‑of‑Experts language models that accounts for cooperative interactions between experts. Unlike first‑order methods such as REAP, HOPE derives an objective that provably bounds pruning error and is shown to outperform baselines across three large MoE models, multiple calibration sets, and diverse benchmarks, especially at high pruning rates and on agentic tasks. The results demonstrate that preserving expert interactions allows aggressive compression with minimal performance loss on complex workloads.

By Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia, Stefano Soatto
arXiv AI
Sep 7

When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models

The paper examines how expert pruning—removing low‑importance experts in Mixture‑of‑Experts models—fails when the router is over‑dispersed, a condition caused by aggressive load‑balancing that spreads tokens nearly uniformly across experts. In this regime, traditional importance signals from router probabilities collapse, making perplexity an unreliable predictor of downstream accuracy; for example, the lowest‑perplexity pruning on gpt‑oss‑20B harms mathematical reasoning while the highest‑perplexity pruning preserves it. To address this, the authors introduce Minimax Expert Score Allocation (MESA), a domain‑aware method that iteratively boosts scores for the most affected domain, achieving minimal worst‑case degradation across domains and outperforming baseline pruning strategies on multiple benchmarks while reducing memory usage.

By Berkcan Kapusuzoglu, Connor Pryor, Sangwoo Cho, Supriyo Chakraborty, Shi-Xiong Zhang, Sambit Sahu, Milind Naphade
arXiv Machine Learning
Jun 17

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.

By Zongfang Liu, Guangyi Chen, Shengkun Tang, Yifan Shen, Huan Wang, Xin Yuan
arXiv Machine Learning
Aug 27

Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing

The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.

By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
arXiv Machine Learning
Nov 2

How to Score Experts for One-Shot MoE Expert Pruning: A Systematic Formulation and Selection Principle

The paper proposes a systematic framework for one‑shot pruning of experts in Mixture‑of‑Experts language models, focusing on routing frequency, gate weighting, and activation strength. It derives a selection principle that recommends task‑agnostic pruning use activation‑based criteria without gate information, while task‑specific pruning can incorporate routing and gate data. Two new task‑agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN), are introduced and shown to outperform existing methods across multiple MoE models and benchmarks, improving average performance by up to 8.8 points.

By Zongfang Liu, Jinghui Zhang, Zijian Ma, Guangyi Chen, Xin Yuan
arXiv AI
Oct 5

Dynamic Expert Pruning for Multi-Agent Systems

Dynamic Expert Pruning for Multi-Agent Systems introduces DEP, a method that generates a per-request mask of experts based on an agent’s system and task prompts. Unlike static pruning, DEP uses a lightweight predictor trained on workflow transcripts to activate only the experts needed for each specific request, requiring no per-configuration calibration. Experiments across various tasks, models, and MoE architectures show DEP outperforms static pruning and merging baselines, especially when few experts are retained, and it generalizes to unseen workflows.

By Jabin Koo, Soheil Abbasloo, Sungjae Lee, Jungseul Ok