arXiv:2610. 23006v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference.
By Wenhui Lv, Yunming Liao, Hongli Xu, Yang Xu, Junhao Cheng
arXiv:2607. 08782v1 Announce Type: cross Abstract: Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models.
By Qianli Liu, Kaibin Guo, Zicong Hong, Peng Li, Fahao Chen, Haodong Wang, Jian Lin, Song Guo
The paper introduces 3D-MoE, a hardware–software co‑designed system that optimizes Mixture‑of‑Experts (MoE) decoding on 3D‑stacked near‑memory processing (NMP) by exploiting expert complementarity. It groups experts offline, maps them locally to balance load, and uses a lightweight Collaborative Memory‑Network to stream hot expert weights, achieving an average 4.63× speedup over expert‑parallelism and 1.58×–1.86× throughput gains over existing NMP baselines with only a 2.11% area increase.
By Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyang Ai, Haonan Zhu, Mengdi Wang, Yinhe Han, Ying Wang
The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.
By Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
Dynamic Expert Quantization (DynaExq) is a runtime-aware mixed-precision serving system designed for single‑GPU Mixture‑of‑Experts (MoE) inference under a hard high‑bandwidth memory (HBM) envelope. It treats the problem as an online, budget‑constrained precision allocation task, keeping the most frequently used experts at higher precision while relegating the rest to low‑precision fallbacks. By estimating expert hotness from router traces and asynchronously promoting or demoting experts, DynaExq maintains a fully materialized expert set during the forward pass, improving accuracy and throughput compared to static post‑training quantization and offloading/prefetch baselines.
whyItMatters":"DynaExq enables efficient deployment of large MoE models on memory‑limited GPUs by dynamically allocating precision based on runtime expert usage, thereby reducing memory footprint and latency while boosting accuracy and throughput."
By Kexin Chu, Dawei Xiang, Zixu Shen, Yiwei Yang, Zecheng Liu, Wei Zhang
Fuse-All MoE is a TPU kernel that fuses token collection, expert projections, gated activation, and output writes to improve Mixture-of-Experts inference. It keeps intermediate expert computation on chip and uses fixed token-slot destinations to allow dynamic scheduling and expert placement while preserving routing semantics. Experiments show the kernel reduces runtime for skewed workloads and lowers latency, boosting overall serving throughput by up to 34.5% compared to the Megablox EP16 baseline.
By Rupeng Liu, Bangsheng Tang, Zongwei Zhou, Chengji Yao, Yejin Lee
Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrain...
ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.
By Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin, Jaejin Lee
Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with traini...
arXiv:2609.38090v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...
By Sanjali Yadav, Bahar Asgari
arXiv:2606. 15453v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.
By Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth, Ke Wang
PCoMoE introduces a path‑compositional execution framework that moves Mixture‑of‑Experts inference from coarse‑grained expert selection to fine‑grained path composition. It uses a path‑level formulation of expert computation, a compatibility‑aware layer‑wise pruning strategy to eliminate low‑value path combinations, and a hardware‑friendly execution engine that reuses sub‑expert structures with bounded overhead. Experiments show up to a 1.31× speedup and a 10% accuracy improvement over existing MoE inference methods.
By Ziyan Gan, Fangxin Liu, Chenyang Guan, Junjie Wang, Ning Yang, Haomin Li, Xiang Li, Siran Yang, Jiamang Wang, Lin Qu, Zongwu Wang, Li Jiang, Haibing Guan