← Back to all news
arXiv AI By Gwen Legate, Rahaf Aljundi, Eugene Belilovsky

Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets

Read the original on arXiv AI →

Optimal Hydra is a transformer‑based heterogeneous federated learning framework designed for fine‑tuning large language models. It creates client‑specific sub‑networks, called Hydra‑nets, by selecting global layers and tailoring attention heads to each client’s local data distribution. Experiments on three datasets demonstrate significant performance gains over baseline methods while reducing local resource demands.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

Task-Centric Personalized Federated Fine-Tuning of Language Models

The paper introduces FedRouter, a personalized federated learning framework that clusters tasks rather than clients to build specialized language models. It employs local and global clustering of adapters to associate them with specific tasks and includes an evaluation router that directs test samples to the most suitable adapter. Experiments on a multitask dataset show FedRouter outperforms existing methods, achieving up to 6.1% relative improvement under task interference and 136% under generalization scenarios.

By Gabriel U. Talasso, Meghdad Kurmanji, Allan M. de Souza, Nicholas D. Lane, Leandro A. Villas
llmsfine-tuning
More like this →
arXiv Machine Learning
Sep 2

FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

FlexP-SFT introduces an aggregation-free framework for personalized split federated fine-tuning of large language models, eliminating the client-side aggregation step that traditionally causes communication bottlenecks and straggler issues. The method employs a layer‑flexible alignment strategy to balance personalization and generalization without global synchronization, and formulates split‑ratio selection as a resource‑aware discrete optimization problem. Experiments demonstrate that FlexP-SFT improves both accuracy and latency compared to baselines, achieving a superior resource‑accuracy trade‑off.

By Jiaxiang Geng, Tianjun Yuan, Pengchao Han, Ying Gao, Xianhao Chen, Bing Luo
llmsfine-tuningsafety
More like this →