arXiv Machine Learning July 3, 2026 By Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

Read the original on arXiv Machine Learning →

arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 24

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

arXiv:2508. 15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.

By Amir Sarfi, Benjamin Th\'erien, Joel Lidin, Eugene Belilovsky

llms efficiency

arXiv AI

Jun 12

MiniMax Sparse Attention

arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.

By Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, Pengyu Zhao

llms agents efficiency multimodal

arXiv Machine Learning

Jun 4

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

arXiv:2606. 04511v1 Announce Type: cross Abstract: Sparse attention reduces compute and memory bandwidth for long-context LLM inference.

By Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, Oreste Villa

llms efficiency

arXiv AI

Jun 19

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

arXiv:2602. 04396v2 Announce Type: replace-cross Abstract: Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth.

By Andrej Jovanovi\'c, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane

arXiv Machine Learning

Jun 10

AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping

arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.

By Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen

llms

arXiv AI

Jun 16

Communication-Efficient Verifiable Attention for LLM Inference

arXiv:2606. 16352v1 Announce Type: cross Abstract: Computation integrity of remote large language model (LLM) serving can be questionable.

By Ziqun Chen, Ming Wu, Michael Heinrich, Jason Zeng, Huiying Lan, Tianwei Zhang, Rui Tan

llms

Related stories

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

MiniMax Sparse Attention

SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping

Communication-Efficient Verifiable Attention for LLM Inference