The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Conventional decentralized federated learning (DFL) often focuses on clients, with each client maintaining a model copy, performing updates individually, and undertaking model exchange and integration...
arXiv:2608. 09208v1 Announce Type: cross Abstract: Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL).
Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL). However, DFL suffers from convergence inefficiency under data heterogeneity due to the use of a uniform learning rate (LR) that ignores layer-specific optimization needs.
The paper introduces FedSWE, a federated learning algorithm designed to handle non‑stationary and heterogeneous client availability without requiring prior real‑time knowledge of which devices are online. FedSWE compensates for missed computations, stabilizes global updates, and mixes local updates through implicit gossiping, all while adding only modest memory and computational overhead. The authors prove that FedSWE converges to a stationary point for non‑convex objectives and achieves linear speedup in certain scenarios, and they validate these claims with experiments on real‑world datasets featuring diverse client unavailability patterns.
arXiv:2603. 01730v2 Announce Type: replace Abstract: Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks.
The paper proposes two hybrid algorithms, FL+FSDP and FL+HSDP, that combine sharded data parallelism with federated learning-style aggregations to reduce communication overhead in large-scale AI training. By partitioning GPUs into loosely‑coupled federation groups, the methods keep inter‑group traffic minimal while maintaining a bounded global batch size. Experiments on a Llama3.1 8B model trained on 512 A100 GPUs show up to 8.04× faster data processing and 4.48 lower evaluation perplexity compared to conventional sharded DP.