The paper proposes two hybrid algorithms, FL+FSDP and FL+HSDP, that combine sharded data parallelism with federated learning-style aggregations to reduce communication overhead in large-scale AI training. By partitioning GPUs into loosely‑coupled federation groups, the methods keep inter‑group traffic minimal while maintaining a bounded global batch size. Experiments on a Llama3.1 8B model trained on 512 A100 GPUs show up to 8.04× faster data processing and 4.48 lower evaluation perplexity compared to conventional sharded DP.
By Gianluca Mittone, Marco Aldinucci
The paper proposes two hybrid algorithms, FL+FSDP and FL+HSDP, that combine sharded data parallelism with federated learning-style aggregations to reduce communication overhead in large-scale AI training. By forming loosely‑coupled federation groups, the methods keep inter‑group traffic minimal while maintaining a bounded global batch size. Experiments on a Llama3.1 8B model trained on 512 A100 GPUs show up to 8.04× faster data processing and 4.48 lower evaluation perplexity compared to traditional sharded DP approaches.
arXiv:2409. 17754v2 Announce Type: replace-cross Abstract: Federated Learning (FL) emerges as a distributed machine learning approach that addresses privacy concerns by training AI models locally on devices.
By Diego Cajaraville-Aboy, Ana Fern\'andez-Vilas, Rebeca P. D\'iaz-Redondo, Manuel Fern\'andez-Veiga
arXiv:2606. 06687v1 Announce Type: new Abstract: We investigate cluster formation, involving the number and composition of clusters, in decentralized federated learning (FL) with heterogeneous machine learning (ML) optimizers.
By Su Wang, Mung Chiang, H. Vincent Poor
StoCFL is a clustered federated learning framework designed to address Non-IID data and dynamic client participation. It introduces a flexible clustering mechanism that allows arbitrary client participation and accommodates newly joined clients, improving data efficiency and model performance. Experiments on four Non-IID settings and a real-world dataset demonstrate that StoCFL achieves promising cluster results even when the number of clusters is unknown, outperforming baseline approaches across various scenarios.
By Dun Zeng, Xiangjing Hu, Shiyu Liu, Yue Yu, Qifan Wang, Zenglin Xu
The paper introduces FastFederatedLearning (FFL), an open‑source federated learning framework written in C/C++ that prioritizes performance and customizability. FFL allows users to define arbitrary communication graphs between clients and servers, and it consistently outperforms Intel OpenFL with speedups of 2.5x to 3.69x across x86‑64, ARM‑v8, and RISC‑V platforms. Future plans include a Python wrapper and middleware for diverse communication backends, aiming to support dynamic, evolving federations as long‑term services.
By Gianluca Mittone, Robert Birke, Marco Aldinucci