Obaleth is a compiler that incorporates GPU-initiated communication into a kernel’s intermediate representation, enabling co‑optimization of communication and computation. It generates specialized transport code for NVIDIA and AMD GPUs, optimizing buffer lifetimes, memory ordering, synchronization, and batching. Benchmarks show significant speedups: up to 3.3× faster small‑message sends on H100 GPUs, 1.25×–2.3× faster all‑to‑all on 256 GPUs compared to NVSHMEM and NCCL GIN, and 1.29× faster Qwen3‑235B inference on 16 B200 GPUs in vLLM.
By Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem Unat
arXiv:2606. 09200v1 Announce Type: cross Abstract: The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems.
By Minyu Cui, Miquel Pericas
arXiv:2609.36070v1 Announce Type: cross
Abstract: AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single...
By Stuart H. Sul, Nash Brown, Henry Wildermuth, William Lin, Federico Cassano, Christopher R\'e
arXiv:2606. 30497v1 Announce Type: cross Abstract: We present a comparative study of CUDA optimization strategies applied to forward and backward propagation in a shallow neural network.
By Rania Zitouni, Nadine Bousdjira, Sarah Hasnaoui, Amel Sadoun, Fatma Salhi
Meta’s paper details how the company optimizes AI training on AMD GPUs by co‑designing hardware and models, using an analytical performance model for kernel selection, and extending agentic workflows for microarchitecture‑aware optimizations. The authors compare AMD Instinct MI350X to NVIDIA B200 on a recommendation model, achieving up to 1.5× higher training throughput. Their methods, the first hyperscaler report on productionizing AMD GPUs for large‑scale training, have been incorporated into open‑source PyTorch, Triton, and ROCm.
By Haoyu Zhang, Arshad Rahman, Adnan Aziz, Ajith Prabhudev, Amin Farmahini, Ben Rhoads, Benson Ma, Brian Shi, Cen Zhao, CK Luk, Dingming Wu, Emad El-Haraty, Eric Chen, Frank Li, Gary Huang, Gefei Zuo, Haoran Zhang, Hiwot Kassa, Hongyi Zeng, Jack Taylor, Jeremy Hadidjojo, John Donati, Jonathan Sheffield, Julio Machado Silva, Karthik Ganesan, Kent Russell, KR Kishore, Lei Tian, Liu Ke, Muhammad Osama, Peng Chen, Rakesh Hemnani, Ryan Swann, Saleel Kudchadker, Shankar Yanamandram, Shobhit Kanaujia, Shucai Xiao, Srinath Bayareddy, Srinivas Vaidyanathan, Sudharssun Subramanian, Supadchaya Puangpontip, Taylor Robie, Tianhao Huang, Tyler Hart, Viswesh Sankaran, Weijun Jiang, Wenkai Du, Xiaodong Wang, Xiaohu Guo, Yao Rong, Yinbin Ma, Yue Dong, Yuzhen Huang, Yvonne Liu, Zhan Shu, Dan Rabinovitsj, Ajit Mathews, Chunqiang Tang
arXiv:2603. 16428v2 Announce Type: replace-cross Abstract: Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs.
By Ruijia Yang, Zeyi Wen
arXiv:2603. 02376v2 Announce Type: replace-cross Abstract: Computation and communication in distributed LLM training and inference are traditionally optimized in isolation; expert-crafted systems such as DeepEP, FLUX, and TokenWeave show the potential of co-design but require deep systems expertise and hardware-specific tuning; CUCo is an agentic framework that automates compute-communication co-design of CUDA kernels by combining a structured design-space formalization with a correctness-first fast-path agent for reliable baselines and an evolution-driven slow-path agent for high-performance strategies, achieving up to 1.
By Yoga Sri Varshan Varadharajan, Bodun Hu, Saurabh Agarwal, Aditya Akella
arXiv:2402. 09589v2 Announce Type: replace-cross Abstract: We present MLCC, a novel technique to augment today's congestion control algorithms to accelerate DNN training jobs in shared GPU clusters in a fully distributed manner.
By Anton A. Zabreyko, Sanjoli Narang, Sudarsanan Rajasekaran, Manya Ghobadi
arXiv:2608.21157v1 Announce Type: cross
Abstract: High-performance GPU kernels underpin modern deep learning and scientific computing. As workloads become increasingly diverse and GPU hardware evolve...
By Jinghao Wang, Qiqi Gu, Chenpeng Wu, Jianguo Yao, Haibing Guan, Xijun Li
arXiv:2609.13585v1 Announce Type: cross
Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...
By Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
arXiv:2605.21187v2 Announce Type: replace-cross
Abstract: As distributed model training scales to span hundreds of thousands of GPUs, scale-out networks face unprecedented performance and efficiency...
By Sajy Khashab, Albert Gran Alcoz, Alon Gal, Jacky Romano, Rani Abboud, Yonatan Piasetzky, Lior Maman, Amit Nishry, Barak Gafni, Omer Shabtai, Matty Kadosh, Dror Goldenberg, Gilad Shainer, Mark Silberstein
arXiv:2608. 05033v1 Announce Type: cross Abstract: Sparse matrix kernels are fundamental to scientific computing, graph analytics, and machine learning.
By Shiyang Li, Guangyan Sun, Jinwei Tang, Yanzhi Wang, Mingyi Hong, Caiwen Ding