arXiv:2606. 11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links.
By Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon
RW-LoRA introduces a random‑walk approach to fine‑tune LoRA models in a decentralized setting, using a single model token that moves through the network and updates locally. This eliminates the need for global synchronization and reduces communication and computation costs compared to centralized or gossip‑based methods. The authors provide convergence guarantees for non‑convex objectives and demonstrate competitive performance on NLP tasks across various graph topologies.
By Xingran Chen, Rohit Bhagat, Ghadir Ayache, Rawad Bitar, Yanmin Gong, Salim El Rouayheb
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives.
arXiv:2504. 17471v2 Announce Type: replace-cross Abstract: Gossip Learning (GL) is a decentralized learning paradigm where users iteratively exchange and aggregate models with a small set of neighboring peers.
By Yacine Belal, Mohamed Maouche, Sonia Ben Mokhtar
arXiv:2608. 06563v1 Announce Type: new Abstract: Machine learning and optimization have advanced together, with practical demands motivating new theory and theoretical breakthroughs enabling new applications.
By Grigory Malinovsky
arXiv:2606. 07496v1 Announce Type: new Abstract: Decentralized stochastic optimization is a fundamental paradigm for large-scale learning over networks, where agents communicate only with their neighbors and no central coordinator is required.
By Ming Sun, Kun Yuan