The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.
By Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with traini...
arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.
By Minyu Cui, Anna Wingkvist, Morgan Ericsson
arXiv:2609.13585v1 Announce Type: cross
Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...
By Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.
By Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob, Andrej Jovanovic, Yan Gao, Wanru Zhao, Nicholas D. Lane
arXiv:2607. 01844v1 Announce Type: cross Abstract: This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.
By Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Semih Yavuz, Silvio Savarese, Shafiq Joty
ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.
By Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin, Jaejin Lee
Obaleth is a compiler that incorporates GPU-initiated communication into a kernel’s intermediate representation, enabling co‑optimization of communication and computation. It generates specialized transport code for NVIDIA and AMD GPUs, optimizing buffer lifetimes, memory ordering, synchronization, and batching. Benchmarks show significant speedups: up to 3.3× faster small‑message sends on H100 GPUs, 1.25×–2.3× faster all‑to‑all on 256 GPUs compared to NVSHMEM and NCCL GIN, and 1.29× faster Qwen3‑235B inference on 16 B200 GPUs in vLLM.
By Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat
The paper introduces CoMoE, a communication‑efficient Mixture‑of‑Experts inference system designed for commodity GPUs. By using host‑centric routing, host‑backed token multicast, and fine‑grained token‑level aggregation, CoMoE reduces communication volume and eliminates global synchronization stalls. Experiments on RTX 5090 GPUs show up to 1.46× throughput improvement, approaching NVLink‑capable A800 performance while costing only 23.4% of the hardware.
By Ruwen Fan (Jimmy), Yuezhi Zu (Jimmy), Junru Li (Jimmy), Qingda Hu (Jimmy), Xinjun (Jimmy), Yang, Jiwu Shu, Youyou Lu
Meta’s paper details how the company optimizes AI training on AMD GPUs by co‑designing hardware and models, using an analytical performance model for kernel selection, and extending agentic workflows for microarchitecture‑aware optimizations. The authors compare AMD Instinct MI350X to NVIDIA B200 on a recommendation model, achieving up to 1.5× higher training throughput. Their methods, the first hyperscaler report on productionizing AMD GPUs for large‑scale training, have been incorporated into open‑source PyTorch, Triton, and ROCm.
By Haoyu Zhang, Arshad Rahman, Adnan Aziz, Ajith Prabhudev, Amin Farmahini, Ben Rhoads, Benson Ma, Brian Shi, Cen Zhao, CK Luk, Dingming Wu, Emad El-Haraty, Eric Chen, Frank Li, Gary Huang, Gefei Zuo, Haoran Zhang, Hiwot Kassa, Hongyi Zeng, Jack Taylor, Jeremy Hadidjojo, John Donati, Jonathan Sheffield, Julio Machado Silva, Karthik Ganesan, Kent Russell, KR Kishore, Lei Tian, Liu Ke, Muhammad Osama, Peng Chen, Rakesh Hemnani, Ryan Swann, Saleel Kudchadker, Shankar Yanamandram, Shobhit Kanaujia, Shucai Xiao, Srinath Bayareddy, Srinivas Vaidyanathan, Sudharssun Subramanian, Supadchaya Puangpontip, Taylor Robie, Tianhao Huang, Tyler Hart, Viswesh Sankaran, Weijun Jiang, Wenkai Du, Xiaodong Wang, Xiaohu Guo, Yao Rong, Yinbin Ma, Yue Dong, Yuzhen Huang, Yvonne Liu, Zhan Shu, Dan Rabinovitsj, Ajit Mathews, Chunqiang Tang
arXiv:2609.38090v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...
By Sanjali Yadav, Bahar Asgari
arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.
By Wenxin Wang, Yule Hou, Yu Ji, Peng Qu, Youhui Zhang