AdaBoosting Text Prompts for Vision-Language Models
arXiv:2607. 00684v1 Announce Type: new Abstract: The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts.
ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.
arXiv:2607. 00684v1 Announce Type: new Abstract: The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts.
arXiv:2602. 21397v2 Announce Type: replace-cross Abstract: Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights.
arXiv:2609.36680v1 Announce Type: new Abstract: Visual reprogramming adapts pretrained models to downstream tasks by modifying their input and output interfaces while keeping the backbone fixed. In v...
The paper introduces Language-driven Dense Semantic Adaptor (LDSA) for multi-label image classification with incomplete annotations. LDSA leverages multimodal pretrained CLIP models to extract prior-adaptive relationships, employing a densely contrastive adaptor for visual contrastive constraints and a language-driven interactive decoder with class-specific prompt tuning. Experiments show LDSA achieves state‑of‑the‑art performance on public benchmarks and reveals implicit semantic relationships through its learning scheme.
Prompt learning modifies vision‑language models by optimizing continuous prompt vectors, yet the resulting prompts are hard to interpret in natural language. PromptSpLiCE is a post‑hoc method that rewrites each class‑conditioned text embedding as a sparse mix of concepts from a fixed dictionary, enabling a direct comparison of concept profiles before and after prompt learning. Across 11 image‑classification datasets, the method shows that only about 1.6 of the initial top‑10 concepts remain after learning, and that larger profile changes correlate with higher accuracy gains, while a derived gradient expression offers geometric insight into loss sensitivity.
arXiv:2610.07572v1 Announce Type: cross Abstract: In-context learning (ICL) adapts frozen large multimodal models (LMMs) to new tasks from a few demonstrations (demos), but re-encodes them at every q...
ES‑VP introduces Energy‑Shaped Visual Prompting, a method that generates image‑specific prompts through low‑rank initialization and energy‑guided dynamic adaptation. It achieves higher performance than existing single‑prompt and diverse‑prompt approaches while using far fewer parameters. Experiments on five architectures and fifteen datasets show consistent superiority, including a 2.6% accuracy gain over DAM‑VP on CLIP with 590× fewer prompt parameters.
The paper introduces EvoPrompt, a framework for adapting vision‑language models to new tasks with limited data while preventing catastrophic forgetting. EvoPrompt uses a Modality‑Shared Prompt Projector to create hierarchical prompts and an evolutionary training strategy that separates low‑rank updates into directional and magnitude components, preserving learned semantic directions. Experiments show that EvoPrompt achieves state‑of‑the‑art few‑shot performance while maintaining the original zero‑shot capabilities of the pre‑trained models.
Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.
arXiv:2607. 28967v1 Announce Type: cross Abstract: Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch.
arXiv:2505.07675v3 Announce Type: replace-cross Abstract: Semi-supervised learning (SSL) has emerged as a practical solution for addressing data scarcity challenges by leveraging unlabeled data. Rece...
TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.