arXiv AI June 3, 2026 By Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohi Dibua, Yifan Gong, Yan Kang, Dimitris N. Metaxas

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

Read the original on arXiv AI →

arXiv:2512. 13996v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Aug 12

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

arXiv:2608. 10605v1 Announce Type: cross Abstract: In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages.

By Soumajyoti Sarkar, Yuxin Tang, Sheng Zha

llms efficiency

arXiv AI

Aug 5

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

arXiv:2608. 03457v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood.

By Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou, Zhenzhong Lan, Yankai Lin, Wayne Xin Zhao, Jianguo Li, Chongxuan Li, Ji-Rong Wen

llms diffusion fine-tuning benchmarks

arXiv AI

Jun 2

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.

By Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng

benchmarks

arXiv AI

Jun 17

SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

arXiv:2606. 17952v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing.

By Miko{\l}aj Zasada, {\L}ukasz Struski, Jacek Tabor, Marcin Kurdziel

llms

arXiv AI

Jul 15

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

arXiv:2607. 12696v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns.

By Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling, Hanbin Dai, Yu Zheng, Wen Hu

llms benchmarks

arXiv Machine Learning

Jun 17

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.

By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath

llms benchmarks

Related stories

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models