arXiv Machine Learning
Jun 8
Sparsely gated tiny linear experts
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
arXiv:2606. 09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts.
Related stories
arXiv Machine Learning
Aug 10
The Sparsity Whisperer
arXiv:2608. 06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs.
arXiv AI
Jul 28
TriSP: Tri-Signal Structured Pruning for Large Language Models
arXiv:2607. 22587v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but their deployment is constrained by the memory and compute cost of their parameters.
arXiv Machine Learning
Jun 29
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
arXiv Machine Learning
Jun 17
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.
arXiv Machine Learning
Jul 21
NIRVANA: Structured Pruning Reimagined for Large Language Model Compression
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.
arXiv AI
Jun 16
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
arXiv:2606. 16825v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory.
arXiv AI
Jul 7
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.
arXiv Machine Learning
Aug 4
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
arXiv:2603. 13418v2 Announce Type: replace Abstract: Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated.
arXiv AI
Jun 9
Pruning and Distilling Mixture-of-Experts into Dense Language Models
arXiv:2605. 28207v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment.
Hugging Face Trending Papers
Jul 2
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns.
arXiv AI
Jun 10
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.