FedFit introduces a federated fine‑tuning framework for large language models that reduces communication overhead by using a disjoint shared vector‑bank parameterization to reconstruct adapter matrices from two compact global vector banks. It resolves the aggregation dilemma between Sum‑of‑Products and Product‑of‑Sums through an alternating optimization schedule that alternates between accurate single‑bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism. The method also incorporates blockwise quantization with client‑side error feedback and provides theoretical convergence guarantees, achieving perplexity comparable to standard federated LoRA while delivering up to 100× higher compression ratios on Qwen2.5 models.
arXiv:2601. 11219v3 Announce Type: replace-cross Abstract: Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank Adaptation (LoRA) are widely adopted to reduce communication and memory costs.
FedSubMuon introduces a communication‑efficient federated fine‑tuning approach for large language models by optimizing compact coefficient matrices within shared structured subspaces, thereby keeping Muon’s matrix‑aware optimization while reducing client upload size. An accuracy‑oriented variant, FedSubMuon‑GT, further adapts subspace bases using projected gradients to better align with task‑relevant directions. Experiments on instruction tuning and mathematical reasoning demonstrate that FedSubMuon‑GT achieves the best overall accuracy on most dataset‑model pairs, while FedSubMuon outperforms all matched‑budget baselines and reduces communication by up to 5.5× on Llama‑1B and 1.4× on Qwen‑4B compared to the closest baseline.
The paper introduces FedRoRA, a federated learning framework that combines Low‑Rank Adaptation (LoRA) with rank‑heterogeneous personalization. It separates model adaptation into shared global directions and client‑specific rank‑wise magnitudes, using SVD on the server to extract a global subspace and a personalized projection with top‑k selection for each client. Experiments on natural language understanding and generation tasks show that FedRoRA outperforms existing state‑of‑the‑art methods.
arXiv:2606. 15963v1 Announce Type: cross Abstract: Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models.
arXiv:2608. 05127v1 Announce Type: cross Abstract: Achieving local differential privacy in distributed optimization while maintaining low communication cost remains challenging.
arXiv:2608. 03605v1 Announce Type: new Abstract: Federated fine-tuning with Low-Rank Adaptation (LoRA) enables efficient collaborative adaptation of Large Language Models (LLMs) without centralizing private data.
FedLore introduces a communication- and memory-efficient federated learning framework that shares a low-rank optimization basis across clients each round, mitigating subspace fragmentation and enabling exact low-rank aggregation. By refreshing this shared basis across rounds, FedLore allows model updates to exceed the per-round rank budget while maintaining a provable $O(T^{-1/2})$ stationarity bound under standard assumptions. Experiments on vision and language tasks, including federated pre‑training, demonstrate that FedLore outperforms low‑rank adapter baselines and matches or surpasses full‑parameter training while reducing communication and optimizer‑state memory.
FlexP-SFT introduces an aggregation-free framework for personalized split federated fine-tuning of large language models, eliminating the client-side aggregation step that traditionally causes communication bottlenecks and straggler issues. The method employs a layer‑flexible alignment strategy to balance personalization and generalization without global synchronization, and formulates split‑ratio selection as a resource‑aware discrete optimization problem. Experiments demonstrate that FlexP-SFT improves both accuracy and latency compared to baselines, achieving a superior resource‑accuracy trade‑off.
arXiv:2511. 19959v2 Announce Type: replace Abstract: Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm.
arXiv:2604. 25421v2 Announce Type: replace-cross Abstract: Federated fine-tuning provides a practical route to adapt large language models (LLMs) on edge devices without centralizing private data, yet in mobile deployments the training wall-clock is often bottlenecked by straggler-limited uplink communication under heterogeneous bandwidth and intermittent participation.
Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability.
arXiv:2607. 14367v1 Announce Type: new Abstract: Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable.