Hugging Face Trending Papers June 4, 2026

Less is MoE: Trimming Experts in Domain-Specialist Language Models

Read the original on Hugging Face Trending Papers →

Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning.

Summary generated by The Flow from the publisher's feed. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning

Jun 5

Less is MoE: Trimming Experts in Domain-Specialist Language Models

arXiv:2606. 05538v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges.

By Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, Heather Miller

llms efficiency benchmarks

arXiv Machine Learning

Jun 17

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.

By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath

llms benchmarks

arXiv AI

Jul 7

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.

By Yushu Zhao, Zheng Wang, Minjia Zhang

llms benchmarks

arXiv AI

Jun 9

Pruning and Distilling Mixture-of-Experts into Dense Language Models

arXiv:2605. 28207v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment.

By Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho

llms efficiency

arXiv Machine Learning

Jun 29

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.

By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You

llms fine-tuning efficiency

arXiv Machine Learning

Jun 17

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.

By Zongfang Liu, Guangyi Chen, Shengkun Tang, Yifan Shen, Huan Wang, Xin Yuan

llms efficiency benchmarks

Related stories

Less is MoE: Trimming Experts in Domain-Specialist Language Models

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Pruning and Distilling Mixture-of-Experts into Dense Language Models

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning