The paper introduces FedRouter, a personalized federated learning framework that clusters tasks rather than clients to build specialized language models. It employs local and global clustering of adapters to associate them with specific tasks and includes an evaluation router that directs test samples to the most suitable adapter. Experiments on a multitask dataset show FedRouter outperforms existing methods, achieving up to 6.1% relative improvement under task interference and 136% under generalization scenarios.
arXiv:2608. 12108v1 Announce Type: new Abstract: Federated learning (FL) enables collaborative model training across distributed clients while keeping data local.
The paper introduces FedPFT, a federated learning framework that tackles the feature‑classifier mismatch problem by using personalized prompts processed through a shared self‑attention transformation module. Unlike prior methods that either degrade the feature extractor or address the mismatch only after training, FedPFT aligns local features with the global classifier during training, improving aggregation and model performance. Experiments demonstrate that FedPFT surpasses state‑of‑the‑art methods by up to 5.07%, and gains up to 7.08% when combined with collaborative contrastive learning.
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
arXiv:2405. 16472v2 Announce Type: replace Abstract: Contemporary AI faces the challenge of balancing generality with user-specific personalization.
arXiv:2502. 08829v2 Announce Type: replace Abstract: Federated learning (FL) with non-IID data often degrades client performance below local training baselines.
The paper introduces FedTAR, a task-aware federated fine‑tuning approach for Mixture‑of‑Experts (MoE) large language models. FedTAR links local client updates to task preferences using routing outputs and Singular Value Decomposition to extract low‑dimensional task coordinates and update directions. It then aggregates updates within and across task clusters, reconstructing the final update to preserve expert specialization and reduce interference, achieving state‑of‑the‑art performance on four benchmark tasks under non‑IID settings.
Optimal Hydra is a transformer‑based heterogeneous federated learning framework designed for fine‑tuning large language models. It creates client‑specific sub‑networks, called Hydra‑nets, by selecting global layers and tailoring attention heads to each client’s local data distribution. Experiments on three datasets demonstrate significant performance gains over baseline methods while reducing local resource demands.
FedRAPP introduces a personalized aggregation framework for federated test‑time prompt tuning of vision‑language models, enabling distributed clients to adapt soft prompts on unlabeled test streams without source data. The method incorporates reliability‑aware client assessment, server‑side exponential moving‑average prompt memory, and similarity‑guided peer aggregation to selectively combine information from reliable and compatible clients. Experiments on nine CLIP benchmark datasets and ImageNet variants show that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while preserving prompt‑only adaptation efficiency.
The paper addresses the mismatch between learner and client data distributions in federated learning, noting that traditional client selection methods often ignore this misalignment. It introduces a dynamic, influence-aware client selection framework that uses a small proxy dataset to estimate each client's utility for the learner’s objective, prioritizing informative sources while mitigating noise and heterogeneity. Experiments on CIFAR-10 with heterogeneous partitions show the proposed method outperforms static and dynamic baselines, achieving faster convergence and higher accuracy.
arXiv:2605. 11165v3 Announce Type: replace Abstract: Federated learning (FL) in heterogeneous environments remains challenging because client models often differ in both architecture and data distribution.
arXiv:2608. 15311v1 Announce Type: new Abstract: Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing.
arXiv:2606. 15625v1 Announce Type: new Abstract: The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation.