FLoKD is an adaptive knowledge‑distillation framework designed for federated fine‑tuning of low‑rank LLMs over wireless networks. It transmits intermediate LoRA activations instead of full parameters or token‑level logits, and uses transformer block importance scoring plus dataset selection to reduce communication. Experiments on WikiText‑103, PTB, and Dialog show a 50‑65% reduction in communication while maintaining competitive perplexity.
By Xinlu Zhang, Na Yan, Yang Su, Yansha Deng, Toktam Mahmoodi
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
By Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su
arXiv:2607. 21074v1 Announce Type: new Abstract: Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup, yet existing aggregation strategies face fundamental limitations.
By Hariharan Ramesh, Jyotikrishna Dass
arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.
By Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao
FedLore introduces a communication- and memory-efficient federated learning framework that shares a low-rank optimization basis across clients each round, mitigating subspace fragmentation and enabling exact low-rank aggregation. By refreshing this shared basis across rounds, FedLore allows model updates to exceed the per-round rank budget while maintaining a provable $O(T^{-1/2})$ stationarity bound under standard assumptions. Experiments on vision and language tasks, including federated pre‑training, demonstrate that FedLore outperforms low‑rank adapter baselines and matches or surpasses full‑parameter training while reducing communication and optimizer‑state memory.
By Junkang Liu
arXiv:2606. 04328v1 Announce Type: cross Abstract: Future wireless networks demand rapid adaptation to highly heterogeneous environments and dynamic task configurations, necessitating a shift from conventional rule-based and optimization-driven radio resource management (RRM) toward artificial intelligence (AI)-driven RRM.
By Fatih Temiz, Shavbo Salehi, Melike Erol-Kantarci
arXiv:2605.07111v3 Announce Type: replace-cross
Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
By Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li, Virginia Smith, Kevin Kuo
arXiv:2606. 31717v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks.
By Shuai Yuan, Sudong Cai, Bingzhi Chen, Shuyuan Zheng, Chuan Xiao, Makoto Onizuka, Rui Mao
arXiv:2607. 22251v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for large language models, but its performance depends strongly on how a fixed rank budget is distributed across Transformer modules.
By Wei Zhang, Xinwu Liu, Yihang Cheng
The paper introduces BQ-LoRA, a low‑rank adaptation framework for large language model agents that consolidates multiple LoRA adapters into a single one. It uses a behavior quotient manifold to balance trajectory updates and a decision‑preserving compression module to keep updates within a fixed rank budget while minimizing distortion of decision distributions. Experiments on AppWorld and BrowseComp‑Plus demonstrate that BQ‑LoRA outperforms standard LoRA and other low‑rank methods, with ablations confirming the benefits of both components.
By Pengyang Zhou, Xiaobin Tu, Zhengxi Liu, Rongkun Xue, Haochen Li, Miancan Liu, Ziyuan Chen, Yinggui Wang, Jinkui Ren, Xiantao Zhang
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang
arXiv:2609.00284v1 Announce Type: cross
Abstract: Emerging 6G wireless networks are expected to operate across diverse deployment scenarios, where variations in network topology, user mobility, traff...
By Fatih Temiz, Shavbo Salehi, Melike Erol-Kantarci