BLEND is a framework for federated vision‑language models that balances personalization and generalization. It uses selective fine‑tuning with global vision and text adapters shared across clients and a personalized vision projection adapter for each client. A novel loss function blends outputs from global and personalized adapters, while an anchor design in the projection heads promotes generalization, leading to superior performance over existing baselines.
arXiv:2609.37033v1 Announce Type: new Abstract: Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but sta...
arXiv:2503. 12999v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated exceptional performance in various multi-modal tasks.
The paper introduces MOSAIC, a method for personalized federated vision‑language models that addresses domain heterogeneity by focusing on class‑specific cross‑domain residuals. It constructs a decision‑aware harmfulness score to identify residuals that hurt image‑text decision margins, then uses a low‑rank residual adapter with shared class factors and private domain factors, an image‑conditioned gate, and harmful‑pair‑aware reweighting to refine local updates. Experiments on Office31, OfficeHome, and DomainNet100 show consistent improvements in macro‑client top‑1 accuracy across various domain‑shift scenarios.
The paper introduces FedGRC, a federated learning framework that enhances both global recognition and personalization for heterogeneous clients. It combines Consensus‑Guided Representation Learning, Multilayer Metric Synthesis, and Support‑Adaptive Shrinkage Personalization to address failures on rare or unseen classes. Experiments on CIFAR‑10, CIFAR‑100, and the FedRS dataset show that FedGRC outperforms 21 baseline methods, improving global accuracy and personalization floors by over 2% in several settings.
arXiv:2602. 14401v2 Announce Type: replace-cross Abstract: Vision-Language Navigation VLN requires large-scale trajectory instruction data from private indoor environments, raising significant privacy concerns.
FedRAPP introduces a personalized aggregation framework for federated test‑time prompt tuning of vision‑language models, enabling distributed clients to adapt soft prompts on unlabeled test streams without source data. The method incorporates reliability‑aware client assessment, server‑side exponential moving‑average prompt memory, and similarity‑guided peer aggregation to selectively combine information from reliable and compatible clients. Experiments on nine CLIP benchmark datasets and ImageNet variants show that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while preserving prompt‑only adaptation efficiency.
The paper introduces FedRouter, a personalized federated learning framework that clusters tasks rather than clients to build specialized language models. It employs local and global clustering of adapters to associate them with specific tasks and includes an evaluation router that directs test samples to the most suitable adapter. Experiments on a multitask dataset show FedRouter outperforms existing methods, achieving up to 6.1% relative improvement under task interference and 136% under generalization scenarios.
FedCKA introduces a Centered Kernel Alignment (CKA)-based method for federated 3D perception that dynamically balances personalization and globalization. By computing layer-wise feature similarities between local client models and a global consensus model, FedCKA generates client‑specific aggregation masks to selectively share representation‑consistent layers. Experiments on a unified multi‑domain nuScenes benchmark demonstrate that FedCKA surpasses established federated baselines, improving average NDS by 7 percentage points.
The paper introduces a training‑free method for personalizing vision‑language models by using calibrated residual decoding. It constructs three evidence conditions—positive, counterfactual, and empty profiles—to isolate the true contribution of personalization. Normalized‑entropy calibration adjusts the influence of this residual signal based on its uncertainty, improving identity‑sensitive visual personalization without fine‑tuning.
arXiv:2610.09907v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information con...
arXiv:2409. 06067v3 Announce Type: replace Abstract: Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients.
arXiv:2605.19750v2 Announce Type: replace Abstract: Visual autoregressive (VAR) models have recently emerged as an efficient paradigm for text-to-image generation, yet their personalization capabilit...