The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.
By Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
arXiv:2606. 15453v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.
By Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth, Ke Wang
arXiv:2607. 12696v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns.
By Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling, Hanbin Dai, Yu Zheng, Wen Hu
arXiv:2609.36222v1 Announce Type: new
Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
By Ali Abbasi, Justin Shi, Soheil Kolouri
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
By Dengke Han