arXiv AI
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 17
Higher-order pruning of experts in mixture-of-experts language models
The paper introduces HOPE, a second‑order pruning method for Mixture‑of‑Experts language models that accounts for cooperative interactions between experts. Unlike first‑order methods such as REAP, HOPE derives an objective that provably bounds pruning error and is shown to outperform baselines across three large MoE models, multiple calibration sets, and diverse benchmarks, especially at high pruning rates and on agentic tasks. The results demonstrate that preserving expert interactions allows aggressive compression with minimal performance loss on complex workloads.
arXiv Machine Learning
Sep 30
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.
arXiv Machine Learning
Jun 17
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.
Hugging Face Trending Papers
Jul 2
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns.
arXiv AI
Jul 3
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
arXiv:2607. 01710v1 Announce Type: new Abstract: Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging.
Hugging Face Trending Papers
Sep 16
Higher-order pruning of experts in mixture-of-experts language models
Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count,...