arXiv Machine Learning June 5, 2026 By Haoze He, Xinkai Zou, Xuan Jiang, Xingyuan Ding, Ao Qu, Juncheng Billy Li, Heather Miller

Less is MoE: Trimming Experts in Domain-Specialist Language Models

Read the original on arXiv Machine Learning →

arXiv:2606. 05538v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Jun 4

Less is MoE: Trimming Experts in Domain-Specialist Language Models

Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning.

llms efficiency benchmarks

arXiv Computation and Language

Sep 10

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

arXiv:2604.23036v2 Announce Type: replace-cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...

By Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller

fine-tuning benchmarks safety

arXiv Machine Learning

Jun 17

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.

By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath

llms benchmarks

arXiv AI

Jul 7

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.

By Yushu Zhao, Zheng Wang, Minjia Zhang

llms benchmarks

arXiv AI

Jun 9

Pruning and Distilling Mixture-of-Experts into Dense Language Models

arXiv:2605. 28207v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment.

By Junhyuck Kim, Jihun Yun, Haechan Kim, Gyeongman Kim, Joonghyun Bae, Jaewoong Cho

llms efficiency

arXiv Machine Learning

Jun 29

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.

By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Less is MoE: Trimming Experts in Domain-Specialist Language Models

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Pruning and Distilling Mixture-of-Experts into Dense Language Models

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models