arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.
By Minyu Cui, Anna Wingkvist, Morgan Ericsson
Expert parallelism (EP) shards the experts of a Mixture-of-Experts (MoE) layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with traini...
The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.
By Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
arXiv:2607. 28633v1 Announce Type: cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly.
By Sanjeev Rao Ganjihal
arXiv:2604. 23466v2 Announce Type: replace Abstract: NVIDIA's CUDA Tile (CuTile) introduces a Python-based, tile-centric abstraction for GPU kernel development that aims to simplify programming while retaining Tensor Core and Tensor Memory Accelerator (TMA) efficiency on modern GPUs.
By Divakar Kumar Yadav, Tian Zhao, Deepak Kumar
arXiv:2511. 04791v2 Announce Type: replace Abstract: Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases.
By Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Mark Hill, Murali Annavaram
arXiv:2610.07098v1 Announce Type: cross
Abstract: Large transformer-based models increasingly depend on multi-GPU execution, which requires frequent collective communication among GPUs. Existing comm...
By Keyvan Dadashzadeh, Yuehong Zhou, Minyu Cui, Miquel Pericas
ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.
By Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin, Jaejin Lee
arXiv:2512. 20968v2 Announce Type: replace-cross Abstract: Distributed attention is essential for scaling large language models (LLMs) to long contexts, yet existing methods either have limited parallelism or incur high communication costs.
By Sirui Chen, Jingji Chen, Siqi Zhu, Ziheng Jiang, Yanghua Peng, Xuehai Qian
arXiv:2609.36070v1 Announce Type: cross
Abstract: AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single...
By Stuart H. Sul, Nash Brown, Henry Wildermuth, William Lin, Federico Cassano, Christopher R\'e
arXiv:2609.36954v1 Announce Type: cross
Abstract: Distributed inference depends on GPU collective communication that must keep pace with evolving hardware and specialized workloads. However, existing...
By Osayamen Jonathan Aimuyo, Swapnil Gandhi, Christos Kozyrakis
arXiv:2512. 22219v2 Announce Type: replace-cross Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel.
By Xinhao Cheng, Zhihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, Hongyi Jin, Bohan Hou, Mengdi Wu, Yixin Dong, Anthony Yip, Zihao Ye, Songting Wang, Wenqin Yang, Xupeng Miao, Tianqi Chen, Zhihao Jia