arXiv Machine Learning July 3, 2026 By Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

Read the original on arXiv Machine Learning →

arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 24

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

arXiv:2508. 15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.

By Amir Sarfi, Benjamin Th\'erien, Joel Lidin, Eugene Belilovsky

llms efficiency

arXiv Machine Learning

Sep 10

NoLoCo: No-all-reduce Low Communication Training Method for Large Models

arXiv:2506.10911v2 Announce Type: replace Abstract: Training large language models is generally done on clusters containing thousands of accelerators, communicating over a high-bandwidth interconnect...

By Jari Kolehmainen, Nikolay Blagoev, Semih Kara, John Donaghy, Christopher Nies, O\u{g}uzhan Ersoy

llms

arXiv Machine Learning

4d ago

Scaling Zero-Order Pretraining through Model Sharding

arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...

By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e

arXiv AI

Jun 12

MiniMax Sparse Attention

arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.

By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao

llms agents efficiency multimodal

arXiv Machine Learning

Jun 4

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.

By Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa

llms efficiency

arXiv Machine Learning

Sep 15

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

The paper introduces a communication‑efficient method for adapting large language models on decentralized GPU meshes. It proposes an asynchronous two‑circuit system that uses fast compressed training with activation masking for pipeline‑parallel transfer and compressed data‑parallel synchronization, while a slower anchor circuit performs occasional unmasked passes. A spectral correction optimizer then denoises the masked gradients using these anchor priors, enabling high compression rates and achieving up to 40× throughput gains over internet‑grade connections while matching dense uncompressed performance.

By Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Chamin P Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko, James Snewin, Karol Pajak, Harry Xi, Alexander Long

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

NoLoCo: No-all-reduce Low Communication Training Method for Large Models

Scaling Zero-Order Pretraining through Model Sharding

MiniMax Sparse Attention

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes