arXiv AI November 5, 2026 By Gengyu Rao, Ghadeer Ahmed H Alabandi, Su Ann Chong, Thomas Huber, Kailash Gogineni, Subhajit Dutta Chowdhury, Saleel Kudchadker, Suchita Pati, Zirui Zhang, Zhen Huang, Yanyuan Qin, Xiaoming Peng, Lihuan Zhang, Botao Hu, Wei Huang, Yang Bo, Wen Xie, Xiangyu Zhao, Ruibin Zhang, Xiaobo Chen, Vidushi Goyal, Anshu Raina, Claire Songhyun Lee, Jiakun Yan, Yao Fu, Nusrat Islam, Ravi Dwivedula, Jie Zhang, Maria Garzaran, Zhenyu Gu, Emad Barsoum

Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs

Read the original on arXiv AI →

The paper presents a hierarchical data pipelining approach for large language model (LLM) training on AMD GPUs, focusing on reducing idle compute time caused by data unavailability. It leverages three key hardware features—Tensor Data Mover (TDM), transpose‑load instructions, and System/Remote DMA (SDMA/RDMA)—to move and rearrange tensor data efficiently between memory and compute units, and between GPUs. Kernel‑level experiments show average speedups of 1.88× for TDM, 1.19× for transpose‑load, and 1.16×–1.33× for overlapped communication, and the implementation is integrated into open‑source AMD frameworks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv Machine Learning

Jun 30

DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures

arXiv:2511. 15503v3 Announce Type: replace-cross Abstract: High-performance Host processors can integrate Processing-In-Memory (PIM) devices, which can accelerate memory-intensive kernels of Machine Learning (ML) models, including Large Language Models (LLMs), by leveraging the large memory bandwidth available at PIM cores.

By Peiming Yang, Sankeerth Durvasula, Ivan Fernandez, Mohammad Sadrosadati, Onur Mutlu, Gennady Pekhimenko, Christina Giannoula

llms

arXiv Machine Learning

Sep 15

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

arXiv:2609.13592v1 Announce Type: cross Abstract: GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-...

By Anish Saxena, Jae Hyung Ju, Hritvik Taneja, Po-An Tsai, Aamer Jaleel, Christos Kozyrakis, Moinuddin Qureshi

llms efficiency

arXiv AI

Jul 14

Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

arXiv:2607. 10183v1 Announce Type: cross Abstract: Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory.

By Yangyijian Liu, Hongyi Ye, Mingyang Li, Wu-jun Li

llms efficiency

arXiv Machine Learning

Jul 10

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

arXiv:2607. 07862v1 Announce Type: cross Abstract: The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures.

By Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das, Sanjay Patel, Volodymyr Kindratenko

llms efficiency

arXiv AI

Jul 7

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.

By Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral

llms efficiency benchmarks

arXiv AI

Jul 23

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.

By Minyu Cui, Anna Wingkvist, Morgan Ericsson

llms benchmarks

One email a morning, machine-written

Related stories

DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts