ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.
By Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
arXiv:2409. 06067v3 Announce Type: replace Abstract: Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients.
By Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, Hai Li
arXiv:2603. 09493v2 Announce Type: replace-cross Abstract: The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge.
By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li
FedRAPP introduces a personalized aggregation framework for federated test‑time prompt tuning of vision‑language models, enabling distributed clients to adapt soft prompts on unlabeled test streams without source data. The method incorporates reliability‑aware client assessment, server‑side exponential moving‑average prompt memory, and similarity‑guided peer aggregation to selectively combine information from reliable and compatible clients. Experiments on nine CLIP benchmark datasets and ImageNet variants show that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while preserving prompt‑only adaptation efficiency.
By Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali, AKM Mahbubur Rahman
The paper introduces FedPFT, a federated learning framework that tackles the feature‑classifier mismatch problem by using personalized prompts processed through a shared self‑attention transformation module. Unlike prior methods that either degrade the feature extractor or address the mismatch only after training, FedPFT aligns local features with the global classifier during training, improving aggregation and model performance. Experiments demonstrate that FedPFT surpasses state‑of‑the‑art methods by up to 5.07%, and gains up to 7.08% when combined with collaborative contrastive learning.
By Xinghao Wu, Xuefeng Liu, Jianwei Niu, Guogang Zhu, Mingjia Shi, Shaojie Tang, Jing Yuan
The paper introduces MOSAIC, a method for personalized federated vision‑language models that addresses domain heterogeneity by focusing on class‑specific cross‑domain residuals. It constructs a decision‑aware harmfulness score to identify residuals that hurt image‑text decision margins, then uses a low‑rank residual adapter with shared class factors and private domain factors, an image‑conditioned gate, and harmful‑pair‑aware reweighting to refine local updates. Experiments on Office31, OfficeHome, and DomainNet100 show consistent improvements in macro‑client top‑1 accuracy across various domain‑shift scenarios.
By Wentao Yue, Qingyu Mao, Tianyou Lai, Ahmed M. Abdelmoniem, Qilei Li
arXiv:2610.09907v1 Announce Type: new
Abstract: Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information con...
By Ankita Das, Ambarish Parthasarathy, Sumohana S. Channappayya, C. Krishna Mohan
arXiv:2610.02117v1 Announce Type: cross
Abstract: On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a froze...
By Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris
arXiv:2511. 16107v3 Announce Type: replace-cross Abstract: Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training.
By Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu
arXiv:2602. 21397v2 Announce Type: replace-cross Abstract: Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights.
By Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani
The paper introduces FedGRC, a federated learning framework that enhances both global recognition and personalization for heterogeneous clients. It combines Consensus‑Guided Representation Learning, Multilayer Metric Synthesis, and Support‑Adaptive Shrinkage Personalization to address failures on rare or unseen classes. Experiments on CIFAR‑10, CIFAR‑100, and the FedRS dataset show that FedGRC outperforms 21 baseline methods, improving global accuracy and personalization floors by over 2% in several settings.
By Junyu Lin, Bosong Qi, Pengcheng Shi, Jiayuan Li
The paper introduces EvoPrompt, a framework for adapting vision‑language models to new tasks with limited data while preventing catastrophic forgetting. EvoPrompt uses a Modality‑Shared Prompt Projector to create hierarchical prompts and an evolutionary training strategy that separates low‑rank updates into directional and magnitude components, preserving learned semantic directions. Experiments show that EvoPrompt achieves state‑of‑the‑art few‑shot performance while maintaining the original zero‑shot capabilities of the pre‑trained models.
By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li