The paper proposes two hybrid algorithms, FL+FSDP and FL+HSDP, that combine sharded data parallelism with federated learning-style aggregations to reduce communication overhead in large-scale AI training. By forming loosely‑coupled federation groups, the methods keep inter‑group traffic minimal while maintaining a bounded global batch size. Experiments on a Llama3.1 8B model trained on 512 A100 GPUs show up to 8.04× faster data processing and 4.48 lower evaluation perplexity compared to traditional sharded DP approaches.
The paper introduces FastFederatedLearning (FFL), an open‑source federated learning framework written in C/C++ that prioritizes performance and customizability. FFL allows users to define arbitrary communication graphs between clients and servers, and it consistently outperforms Intel OpenFL with speedups of 2.5x to 3.69x across x86‑64, ARM‑v8, and RISC‑V platforms. Future plans include a Python wrapper and middleware for diverse communication backends, aiming to support dynamic, evolving federations as long‑term services.
By Gianluca Mittone, Robert Birke, Marco Aldinucci
arXiv:2608. 14654v1 Announce Type: cross Abstract: Federated Learning (FL) is a collaborative paradigm that enables multiple devices to train a global model while preserving local data privacy.
By Hai Anh Tran, Cuong Ta, Truong X. Tran
arXiv:2608. 09208v1 Announce Type: cross Abstract: Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL).
By Van Truong Vo, Khoa Nguyen, Taehong Kim
arXiv:2411. 02908v2 Announce Type: replace Abstract: Scaling large language models (LLMs) demands extensive data and computing resources, which are traditionally constrained to data centers by the high-bandwidth requirements of distributed training.
By Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, Nicholas D. Lane
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.
By Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob, Andrej Jovanovic, Yan Gao, Wanru Zhao, Nicholas D. Lane