arXiv:2405. 11667v2 Announce Type: replace Abstract: Local SGD is a popular optimization method in distributed learning, often outperforming other algorithms in practice, including mini-batch SGD.
arXiv:2607. 14731v1 Announce Type: new Abstract: Local SGD, also known as Federated Averaging, is a widely used distributed optimization algorithm.
The paper introduces Batched SGD, a variant that groups online samples into epochs and performs a single update per epoch using a low‑variance gradient estimate. This batching approach allows a straightforward high‑probability analysis without restrictive assumptions or auxiliary sequences, yielding near‑optimal rates for both strongly convex and non‑convex objectives under standard smoothness and sub‑Gaussian noise conditions. The authors also extend the method to federated learning, providing the first high‑probability guarantees with logarithmic communication complexity, linear speedup in the number of agents, and robustness to data heterogeneity.
arXiv:2608. 06563v1 Announce Type: new Abstract: Machine learning and optimization have advanced together, with practical demands motivating new theory and theoretical breakthroughs enabling new applications.
arXiv:2606. 01128v1 Announce Type: new Abstract: Communication overhead is a crucial bottleneck in scalable distributed learning.
arXiv:2604. 24012v3 Announce Type: replace Abstract: Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer from slow convergence and high communication and memory costs in heterogeneous, resource-constrained environments.
arXiv:2302. 09832v4 Announce Type: replace Abstract: In distributed optimization and federated learning, slow and costly communication between parallel devices and the central server constitutes the primary bottleneck.
arXiv:2602. 04396v2 Announce Type: replace-cross Abstract: Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth.
The paper proposes a new federated learning approach called FedALS that reduces communication costs by varying aggregation frequencies across model layers. It derives tighter generalization bounds for one‑round and multi‑round federated learning, linking these bounds to local updates and data heterogeneity. Based on representation‑learning insights, the authors argue that infrequent aggregation of early layers and more frequent aggregation of final layers yields more generalizable models, especially in non‑iid settings, and demonstrate the method’s effectiveness experimentally.
The paper introduces FedSWE, a federated learning algorithm designed to handle non‑stationary and heterogeneous client availability without requiring prior real‑time knowledge of which devices are online. FedSWE compensates for missed computations, stabilizes global updates, and mixes local updates through implicit gossiping, all while adding only modest memory and computational overhead. The authors prove that FedSWE converges to a stationary point for non‑convex objectives and achieves linear speedup in certain scenarios, and they validate these claims with experiments on real‑world datasets featuring diverse client unavailability patterns.
arXiv:2311.03154v3 Announce Type: replace Abstract: There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: (i) parallel FL (PFL), where clients tra...
arXiv:2602. 11557v2 Announce Type: replace Abstract: A variety of widely used optimization methods like SignSGD and Muon can be interpreted as instances of steepest descent under different norm-induced geometries.
arXiv:2606. 07496v1 Announce Type: new Abstract: Decentralized stochastic optimization is a fundamental paradigm for large-scale learning over networks, where agents communicate only with their neighbors and no central coordinator is required.