arXiv AI
AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 10
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
arXiv:2506.10911v2 Announce Type: replace Abstract: Training large language models is generally done on clusters containing thousands of accelerators, communicating over a high-bandwidth interconnect...
arXiv Machine Learning
Jul 24
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
arXiv:2508. 15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.
arXiv AI
Jun 10
Unifying Local Communications and Local Updates for LLM Pretraining
arXiv:2606. 11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links.
arXiv Machine Learning
Jul 3
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.
Hugging Face Trending Papers
Jun 9
Unifying Local Communications and Local Updates for LLM Pretraining
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives.
arXiv AI
Jul 7
Can Model Merging Improve Aggregation in DiLoCo?
arXiv:2607. 03011v1 Announce Type: cross Abstract: Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a broad array of methods having been proposed to tackle this problem.