arXiv Machine Learning

Mixture-of-Kittens: MoE Megakernel for NVL72s

arXiv Machine Learning
6d ago

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.

By Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
arXiv AI
Sep 15

mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

arXiv:2609.13585v1 Announce Type: cross Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...

By Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
arXiv Machine Learning
Oct 2

Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.

By Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin, Jaejin Lee
arXiv Machine Learning
1d ago

Compiling Specialized GPU-Initiated Networking

Obaleth is a compiler that incorporates GPU-initiated communication into a kernel’s intermediate representation, enabling co‑optimization of communication and computation. It generates specialized transport code for NVIDIA and AMD GPUs, optimizing buffer lifetimes, memory ordering, synchronization, and batching. Benchmarks show significant speedups: up to 3.3× faster small‑message sends on H100 GPUs, 1.25×–2.3× faster all‑to‑all on 256 GPUs compared to NVSHMEM and NCCL GIN, and 1.29× faster Qwen3‑235B inference on 16 B200 GPUs in vLLM.

By Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat
arXiv Machine Learning
Oct 8

Democratizing MoE inference on commodity GPUs with CoMoE

The paper introduces CoMoE, a communication‑efficient Mixture‑of‑Experts inference system designed for commodity GPUs. By using host‑centric routing, host‑backed token multicast, and fine‑grained token‑level aggregation, CoMoE reduces communication volume and eliminates global synchronization stalls. Experiments on RTX 5090 GPUs show up to 1.46× throughput improvement, approaching NVLink‑capable A800 performance while costing only 23.4% of the hardware.

By Ruwen Fan (Jimmy), Yuezhi Zu (Jimmy), Junru Li (Jimmy), Qingda Hu (Jimmy), Xinjun (Jimmy), Yang, Jiwu Shu, Youyou Lu
arXiv Machine Learning
Oct 16

Architecting for Hardware Diversity: Meta's End-to-End Co-Design for High-Performance AI Training on AMD GPUs

Meta’s paper details how the company optimizes AI training on AMD GPUs by co‑designing hardware and models, using an analytical performance model for kernel selection, and extending agentic workflows for microarchitecture‑aware optimizations. The authors compare AMD Instinct MI350X to NVIDIA B200 on a recommendation model, achieving up to 1.5× higher training throughput. Their methods, the first hyperscaler report on productionizing AMD GPUs for large‑scale training, have been incorporated into open‑source PyTorch, Triton, and ROCm.

By Haoyu Zhang, Arshad Rahman, Adnan Aziz, Ajith Prabhudev, Amin Farmahini, Ben Rhoads, Benson Ma, Brian Shi, Cen Zhao, CK Luk, Dingming Wu, Emad El-Haraty, Eric Chen, Frank Li, Gary Huang, Gefei Zuo, Haoran Zhang, Hiwot Kassa, Hongyi Zeng, Jack Taylor, Jeremy Hadidjojo, John Donati, Jonathan Sheffield, Julio Machado Silva, Karthik Ganesan, Kent Russell, KR Kishore, Lei Tian, Liu Ke, Muhammad Osama, Peng Chen, Rakesh Hemnani, Ryan Swann, Saleel Kudchadker, Shankar Yanamandram, Shobhit Kanaujia, Shucai Xiao, Srinath Bayareddy, Srinivas Vaidyanathan, Sudharssun Subramanian, Supadchaya Puangpontip, Taylor Robie, Tianhao Huang, Tyler Hart, Viswesh Sankaran, Weijun Jiang, Wenkai Du, Xiaodong Wang, Xiaohu Guo, Yao Rong, Yinbin Ma, Yue Dong, Yuzhen Huang, Yvonne Liu, Zhan Shu, Dan Rabinovitsj, Ajit Mathews, Chunqiang Tang