arXiv Machine Learning
GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 7
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.
arXiv Machine Learning
Sep 30
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.
arXiv Machine Learning
Jun 29
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
arXiv AI
Jun 18
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
arXiv:2606. 18304v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead.
arXiv AI
Oct 21
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
arXiv:2610.27123v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert p...
arXiv AI
Sep 2
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs
Residual sparsification via output importance (PARSER) is a new compression technique for mixture-of-experts large language models that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. By introducing output importance, PARSER measures each residual’s contribution to the final expert output and compresses accordingly. Experiments show that PARSER reduces the accuracy gap to the uncompressed model by 1.41× on Qwen and 1.44× on DeepSeek while achieving the same peak memory reduction.