arXiv Machine Learning

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

Fuse-All MoE is a TPU kernel that fuses token collection, expert projections, gated activation, and output writes to improve Mixture-of-Experts inference. It keeps intermediate expert computation on chip and uses fixed token-slot destinations to allow dynamic scheduling and expert placement while preserving routing semantics. Experiments show the kernel reduces runtime for skewed workloads and lowers latency, boosting overall serving throughput by up to 34.5% compared to the Megablox EP16 baseline.

Hugging Face Trending Papers
Jul 27

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.

arXiv AI
Oct 21

3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP

The paper introduces 3D-MoE, a hardware–software co‑designed system that optimizes Mixture‑of‑Experts (MoE) decoding on 3D‑stacked near‑memory processing (NMP) by exploiting expert complementarity. It groups experts offline, maps them locally to balance load, and uses a lightweight Collaborative Memory‑Network to stream hot expert weights, achieving an average 4.63× speedup over expert‑parallelism and 1.58×–1.86× throughput gains over existing NMP baselines with only a 2.11% area increase.

By Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyang Ai, Haonan Zhu, Mengdi Wang, Yinhe Han, Ying Wang
arXiv Machine Learning
Oct 28

COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

COPE (Composable Offloading Policy Emulator) is a system that predicts the best mixture‑of‑experts (MoE) offloading policy for a given model‑platform configuration before runtime integration. It uses a Policy Matrix to separate expert residency and execution strategy, and a Virtual Heterogeneous Machine to profile combined‑operator schedules and estimate performance from offline routing traces. Compared to baselines, COPE improves fastest‑policy selection accuracy by 24‑100 % and reduces prediction error by 1.7‑6.8×, cutting policy‑specific implementation effort by up to 74 %.

By Hanming Zhang, Erhu Feng