arXiv AI
Jun 19
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
arXiv:2602. 04396v2 Announce Type: replace-cross Abstract: Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth.
FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection
FedLore introduces a communication- and memory-efficient federated learning framework that shares a low-rank optimization basis across clients each round, mitigating subspace fragmentation and enabling exact low-rank aggregation. By refreshing this shared basis across rounds, FedLore allows model updates to exceed the per-round rank budget while maintaining a provable $O(T^{-1/2})$ stationarity bound under standard assumptions. Experiments on vision and language tasks, including federated pre‑training, demonstrate that FedLore outperforms low‑rank adapter baselines and matches or surpasses full‑parameter training while reducing communication and optimizer‑state memory.
Related stories
arXiv Machine Learning
Jun 10
FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
arXiv:2604. 24012v3 Announce Type: replace Abstract: Federated learning enables a population of clients to collaboratively train machine learning models without exchanging their raw data, but standard algorithms such as FedAvg suffer from slow convergence and high communication and memory costs in heterogeneous, resource-constrained environments.
arXiv AI
Aug 25
SplitLite: Low-Rank Residual Compression for Split Learning
arXiv:2608.23018v1 Announce Type: cross Abstract: Federated fine-tuning of on-device large language models (LLMs) faces a significant computing burden. To overcome this limitation, split learning (SL...
arXiv Machine Learning
Jul 24
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
arXiv:2607. 21074v1 Announce Type: new Abstract: Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup, yet existing aggregation strategies face fundamental limitations.
arXiv AI
Jul 10
CoCo-Fed: A Unified Framework for Memory- and Communication-Efficient Federated Learning at the Wireless Edge
arXiv:2601. 00549v2 Announce Type: replace-cross Abstract: The deployment of large-scale neural networks within the Open Radio Access Network (O-RAN) architecture is pivotal for enabling native edge intelligence.
arXiv AI
Jun 10
FedSteer: Taming Extreme Gradient Staleness in Federated Learning with Corrective Projections and Caching
arXiv:2606. 10124v1 Announce Type: cross Abstract: Federated learning (FL) is often subject to aggregation variance if clients do not consistently participate in training rounds.
arXiv AI
Aug 11
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
arXiv:2608. 09742v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.
arXiv AI
Jun 16
PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity
arXiv:2606. 15963v1 Announce Type: cross Abstract: Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models.
arXiv AI
Jun 6
Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models
arXiv:2606. 06154v1 Announce Type: new Abstract: Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning.
arXiv Machine Learning
Jul 17
Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
arXiv:2607. 14367v1 Announce Type: new Abstract: Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable.
arXiv AI
Aug 3
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
arXiv:2607. 29071v1 Announce Type: cross Abstract: Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models.
Hugging Face Trending Papers
Jul 15
Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability.