← Back to all news
arXiv Machine Learning By Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Chamin P Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko, James Snewin, Karol Pajak, Harry Xi, Alexander Long

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

Read the original on arXiv Machine Learning →

The paper introduces a communication‑efficient method for adapting large language models on decentralized GPU meshes. It proposes an asynchronous two‑circuit system that uses fast compressed training with activation masking for pipeline‑parallel transfer and compressed data‑parallel synchronization, while a slower anchor circuit performs occasional unmasked passes. A spectral correction optimizer then denoises the masked gradients using these anchor priors, enabling high compression rates and achieving up to 40× throughput gains over internet‑grade connections while matching dense uncompressed performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Jul 3

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.

By Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
llmsefficiency
More like this →
arXiv Machine Learning
Sep 23

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.

By Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 16

ZenFlow: Enabling Stall-Free Offloading for LLM Training

ZenFlow is an offloaded training framework for large language models that reduces GPU stalls by prioritizing important gradients. It updates critical gradients in-place on the GPU while offloading less important ones to the CPU, overlapping CPU work with GPU computation. The approach leverages spatio‑temporal locality of important gradients to keep identification lightweight, achieving up to 5× speedup, 2× less PCIe traffic, 85% fewer GPU stalls, and lower cloud GPU costs without sacrificing accuracy.

By Tingfeng Lan, Yusen Wu, Bin Ma, Zhaoyuan Su, Rui Yang, Tekin Bicer, Masahiro Tanaka, Olatunji Ruwase, Dong Li, Yue Cheng
llmsfine-tuning
More like this →