arXiv Machine Learning
FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computer Vision
Sep 28
ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models
ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.
arXiv AI
Jul 8
MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
arXiv:2409. 06067v3 Announce Type: replace Abstract: Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients.
arXiv AI
Jun 4
EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation
arXiv:2603. 09493v2 Announce Type: replace-cross Abstract: The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge.
arXiv Computer Vision
Oct 19
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
FedRAPP introduces a personalized aggregation framework for federated test‑time prompt tuning of vision‑language models, enabling distributed clients to adapt soft prompts on unlabeled test streams without source data. The method incorporates reliability‑aware client assessment, server‑side exponential moving‑average prompt memory, and similarity‑guided peer aggregation to selectively combine information from reliable and compatible clients. Experiments on nine CLIP benchmark datasets and ImageNet variants show that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while preserving prompt‑only adaptation efficiency.
arXiv Machine Learning
Sep 22
Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation
The paper introduces FedPFT, a federated learning framework that tackles the feature‑classifier mismatch problem by using personalized prompts processed through a shared self‑attention transformation module. Unlike prior methods that either degrade the feature extractor or address the mismatch only after training, FedPFT aligns local features with the global classifier during training, improving aggregation and model performance. Experiments demonstrate that FedPFT surpasses state‑of‑the‑art methods by up to 5.07%, and gains up to 7.08% when combined with collaborative contrastive learning.
arXiv Computer Vision
Oct 2
Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models
The paper introduces MOSAIC, a method for personalized federated vision‑language models that addresses domain heterogeneity by focusing on class‑specific cross‑domain residuals. It constructs a decision‑aware harmfulness score to identify residuals that hurt image‑text decision margins, then uses a low‑rank residual adapter with shared class factors and private domain factors, an image‑conditioned gate, and harmful‑pair‑aware reweighting to refine local updates. Experiments on Office31, OfficeHome, and DomainNet100 show consistent improvements in macro‑client top‑1 accuracy across various domain‑shift scenarios.