← Back to all news
arXiv Computer Vision By Ryo Kamiya, Hiroshi Kera, Kazuhiko Kawamoto

What Does Prompt Learning Change? -A Natural-Language Concept Analysis of Vision-Language Models

Read the original on arXiv Computer Vision →

Prompt learning modifies vision‑language models by optimizing continuous prompt vectors, yet the resulting prompts are hard to interpret in natural language. PromptSpLiCE is a post‑hoc method that rewrites each class‑conditioned text embedding as a sparse mix of concepts from a fixed dictionary, enabling a direct comparison of concept profiles before and after prompt learning. Across 11 image‑classification datasets, the method shows that only about 1.6 of the initial top‑10 concepts remain after learning, and that larger profile changes correlate with higher accuracy gains, while a derived gradient expression offers geometric insight into loss sensitivity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Sep 28

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.

By Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
llmsmultimodalbenchmarks
More like this →
arXiv AI
Sep 2

Guided Prompt Evolution for Vision-Language Models Adaptation

The paper introduces EvoPrompt, a framework for adapting vision‑language models to new tasks with limited data while preventing catastrophic forgetting. EvoPrompt uses a Modality‑Shared Prompt Projector to create hierarchical prompts and an evolutionary training strategy that separates low‑rank updates into directional and magnitude components, preserving learned semantic directions. Experiments show that EvoPrompt achieves state‑of‑the‑art few‑shot performance while maintaining the original zero‑shot capabilities of the pre‑trained models.

By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li
llmsragfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 9

Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

The paper introduces Diffusion Meta-Prompt (DMP), a framework that learns a distribution of prompts via diffusion models, enabling the synthesis of new prompts from natural language task descriptions without needing original task data or losses. DMP incorporates a test-time steering strategy that uses a best training prompt as a latent anchor to stabilize sampling, improving generalization across classification, retrieval, and text-to-image generation tasks. The method reduces storage and inference costs by over 90% compared to prompt retrieval, achieves up to 2.0% average gains over prior meta-learning methods on composite classification, and provides a theoretical bound on expected task loss.

By Deepak Sridhar, Yi Li, Kartikeya Bhardwaj, Shuangjun Liu, Taotao Jing, Yuan Li, Shuai Zhang, Jiancheng Lyu, Dashan Gao, Nuno Vasconcelos
llmsdiffusion
More like this →