arXiv Machine Learning
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 3
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.
arXiv Machine Learning
Jul 24
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
arXiv:2508. 15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.
arXiv AI
4d ago
AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
arXiv:2609.36662v1 Announce Type: cross Abstract: The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of acc...
arXiv AI
Jun 10
Unifying Local Communications and Local Updates for LLM Pretraining
arXiv:2606. 11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links.
arXiv AI
Jun 18
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.
arXiv AI
Jul 7
Can Model Merging Improve Aggregation in DiLoCo?
arXiv:2607. 03011v1 Announce Type: cross Abstract: Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a broad array of methods having been proposed to tackle this problem.