SeedFlood is a novel decentralized fine‑tuning method for large language models that scales to billions of parameters and hundreds of clients. It leverages the seed‑reconstructible structure of zeroth‑order gradients to reduce message sizes to near‑zero, enabling efficient flooding across the network. Experiments show SeedFlood outperforms standard zeroth‑order baselines in communication efficiency and generalization, and rivals first‑order gossip methods while incurring far less communication cost.
arXiv:2606. 11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links.
RW-LoRA introduces a random‑walk approach to fine‑tune LoRA models in a decentralized setting, using a single model token that moves through the network and updates locally. This eliminates the need for global synchronization and reduces communication and computation costs compared to centralized or gossip‑based methods. The authors provide convergence guarantees for non‑convex objectives and demonstrate competitive performance on NLP tasks across various graph topologies.
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives.
arXiv:2504. 17471v2 Announce Type: replace-cross Abstract: Gossip Learning (GL) is a decentralized learning paradigm where users iteratively exchange and aggregate models with a small set of neighboring peers.
arXiv:2608. 06563v1 Announce Type: new Abstract: Machine learning and optimization have advanced together, with practical demands motivating new theory and theoretical breakthroughs enabling new applications.
arXiv:2606. 07496v1 Announce Type: new Abstract: Decentralized stochastic optimization is a fundamental paradigm for large-scale learning over networks, where agents communicate only with their neighbors and no central coordinator is required.
arXiv:2607. 12172v1 Announce Type: cross Abstract: Decentralized gradient descent (DGD) is widely used for solving distributed optimization problems over networks of agents.
arXiv:2609.07312v1 Announce Type: new Abstract: This paper proposes a robust decentralized personalized federated learning method R-DPFL, that enables clients to reduce the impact of Byzantine attack...
arXiv:2504. 12742v2 Announce Type: replace Abstract: Decentralized Federated Learning (DFL) enables collaborative model training without relying on a central server.
arXiv:2606. 04757v1 Announce Type: cross Abstract: We study decentralized stochastic smooth convex optimization, where $M$ workers minimize an average objective using local stochastic gradients and neighbor-only communication over a fixed gossip network.
arXiv:2606. 10774v2 Announce Type: replace Abstract: Decentralized Federated Learning(DFL) enables collaborative model training across wireless edge nodes, including IoT deployments, autonomous vehicles, UAV swarms, and satellite constellations.
arXiv:2605. 09076v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents increasingly collaborate over peer-to-peer networks to improve their reliability.