← Back to all news
arXiv AI By Xinlu Zhang, Na Yan, Yang Su, Yansha Deng, Toktam Mahmoodi

FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks

Read the original on arXiv AI →

FLoKD is an adaptive knowledge‑distillation framework designed for federated fine‑tuning of low‑rank LLMs over wireless networks. It transmits intermediate LoRA activations instead of full parameters or token‑level logits, and uses transformer block importance scoring plus dataset selection to reduce communication. Experiments on WikiText‑103, PTB, and Dialog show a 50‑65% reduction in communication while maintaining competitive perplexity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 2

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

The paper introduces FedRoRA, a federated learning framework that combines Low‑Rank Adaptation (LoRA) with rank‑heterogeneous personalization. It separates model adaptation into shared global directions and client‑specific rank‑wise magnitudes, using SVD on the server to extract a global subspace and a personalized projection with top‑k selection for each client. Experiments on natural language understanding and generation tasks show that FedRoRA outperforms existing state‑of‑the‑art methods.

By Lei Wang, Jieming Bian, Letian Zhang, Jie Xu
llmsfine-tuningbenchmarkssafety
More like this →
arXiv AI
Jun 16

SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

arXiv:2601. 11219v3 Announce Type: replace-cross Abstract: Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank Adaptation (LoRA) are widely adopted to reduce communication and memory costs.

By Zhikang Shen, Jianrong Lu, Haiyuan Wan, Jianhai Chen
llmsfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 4

Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach

arXiv:2601. 21369v2 Announce Type: replace Abstract: Recent studies of federated graph foundational models (FedGFMs) break the idealized and untenable assumption of having centralized data storage to train graph foundation models, and accommodate the reality of distributed, privacy-restricted data silos.

By Yinlin Zhu, Di Wu, Xianzhi Zhang, Yuming Ai, Xunkai Li, Miao Hu, Guocong Quan
llmsragfine-tuningefficiencysafety
More like this →
arXiv Machine Learning
Jun 16

Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts

arXiv:2606. 15625v1 Announce Type: new Abstract: The continuous scaling of large language models (LLMs) incurs prohibitive computational costs, making Mixture-of-Experts (MoE) a scalable alternative for efficient fine-tuning via sparse activation.

By Yijun Lu, Zihan Fang, Pengpeng Qiao, Zheng Lin, Jing Yang, Yuxin Zhang, Por Lip Yee, Zhe Chen, Jun Luo
llmsfine-tuningsafety
More like this →