Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation.
arXiv:2607. 26618v1 Announce Type: new Abstract: Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples.
The paper introduces FedTAR, a task-aware federated fine‑tuning approach for Mixture‑of‑Experts (MoE) large language models. FedTAR links local client updates to task preferences using routing outputs and Singular Value Decomposition to extract low‑dimensional task coordinates and update directions. It then aggregates updates within and across task clusters, reconstructing the final update to preserve expert specialization and reduce interference, achieving state‑of‑the‑art performance on four benchmark tasks under non‑IID settings.
arXiv:2604.23036v2 Announce Type: replace-cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
arXiv:2606. 15625v1 Announce Type: new Abstract: The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation.
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
arXiv:2608. 15311v1 Announce Type: new Abstract: Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing.
arXiv:2606. 01007v1 Announce Type: cross Abstract: Sparsely activated Mixture-of-Experts (MoE) models scale capacity via conditional computation, but distributed inference suffers from cross-GPU expert communication and routing-induced load imbalance.
The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
arXiv:2608. 13911v1 Announce Type: new Abstract: Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may contain different partial modality subsets.
arXiv:2502. 04411v3 Announce Type: replace-cross Abstract: Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one.
arXiv:2607. 12111v1 Announce Type: cross Abstract: Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges.