arXiv Machine Learning By Rupeng Liu, Bangsheng Tang, Zongwei Zhou, Chengji Yao, Yejin Lee

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

Read the original on arXiv Machine Learning →

Fuse-All MoE is a TPU kernel that fuses token collection, expert projections, gated activation, and output writes to improve Mixture-of-Experts inference. It keeps intermediate expert computation on chip and uses fixed token-slot destinations to allow dynamic scheduling and expert placement while preserving routing semantics. Experiments show the kernel reduces runtime for skewed workloads and lowers latency, boosting overall serving throughput by up to 34.5% compared to the Megablox EP16 baseline.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers
Jul 27

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.