DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
arXiv:2610. 23006v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference.
arXiv:2607. 08782v1 Announce Type: cross Abstract: Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models.
The paper introduces 3D-MoE, a hardware–software co‑designed system that optimizes Mixture‑of‑Experts (MoE) decoding on 3D‑stacked near‑memory processing (NMP) by exploiting expert complementarity. It groups experts offline, maps them locally to balance load, and uses a lightweight Collaborative Memory‑Network to stream hot expert weights, achieving an average 4.63× speedup over expert‑parallelism and 1.58×–1.86× throughput gains over existing NMP baselines with only a 2.11% area increase.
The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.
Dynamic Expert Quantization (DynaExq) is a runtime-aware mixed-precision serving system designed for single‑GPU Mixture‑of‑Experts (MoE) inference under a hard high‑bandwidth memory (HBM) envelope. It treats the problem as an online, budget‑constrained precision allocation task, keeping the most frequently used experts at higher precision while relegating the rest to low‑precision fallbacks. By estimating expert hotness from router traces and asynchronously promoting or demoting experts, DynaExq maintains a fully materialized expert set during the forward pass, improving accuracy and throughput compared to static post‑training quantization and offloading/prefetch baselines. whyItMatters":"DynaExq enables efficient deployment of large MoE models on memory‑limited GPUs by dynamically allocating precision based on runtime expert usage, thereby reducing memory footprint and latency while boosting accuracy and throughput."
Fuse-All MoE is a TPU kernel that fuses token collection, expert projections, gated activation, and output writes to improve Mixture-of-Experts inference. It keeps intermediate expert computation on chip and uses fixed token-slot destinations to allow dynamic scheduling and expert placement while preserving routing semantics. Experiments show the kernel reduces runtime for skewed workloads and lowers latency, boosting overall serving throughput by up to 34.5% compared to the Megablox EP16 baseline.