arXiv AI
Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 23
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
arXiv:2609.25809v1 Announce Type: new Abstract: Fine-grained mixture-of-experts (MoE) architectures have become a mainstream design for open-weight LLMs, with hundreds of experts and increasingly man...
arXiv AI
Jul 3
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
arXiv:2607. 01710v1 Announce Type: new Abstract: Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging.
arXiv Machine Learning
Sep 30
BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding
arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
arXiv Machine Learning
Jun 17
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.
Hugging Face Trending Papers
Jul 2
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns.
arXiv AI
Sep 7
Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
arXiv:2609. 04575v1 Announce Type: cross Abstract: Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities.