arXiv Machine Learning

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

arXiv:2602. 21397v2 Announce Type: replace-cross Abstract: Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights.

arXiv Computer Vision

6d ago

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.

By Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar

llms multimodal benchmarks

arXiv Machine Learning

Jul 2

AdaBoosting Text Prompts for Vision-Language Models

arXiv:2607. 00684v1 Announce Type: new Abstract: The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts.

By Seokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, Jungseul Ok

llms multimodal benchmarks

arXiv AI

Aug 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv Computer Vision

Sep 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning

arXiv:2608.06411v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by...

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 25

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

arXiv:2511. 11421v2 Announce Type: replace-cross Abstract: Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge.

By Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang, Han-Jia Ye, De-Chuan Zhan

llms multimodal benchmarks

arXiv Computation and Language

Sep 22

SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models

arXiv:2508.03351v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-la...

By Yufei Xue, Yushi Huang, Lunjie Zhu, Jiawei Shao, Jun Zhang

llms efficiency multimodal benchmarks

arXiv Computer Vision

Aug 24

ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation

ES‑VP introduces Energy‑Shaped Visual Prompting, a method that generates image‑specific prompts through low‑rank initialization and energy‑guided dynamic adaptation. It achieves higher performance than existing single‑prompt and diverse‑prompt approaches while using far fewer parameters. Experiments on five architectures and fifteen datasets show consistent superiority, including a 2.6% accuracy gain over DAM‑VP on CLIP with 590× fewer prompt parameters.

By Can Jin, Ying Li, Jingchen Sun, Hongwu Peng, Jiahui Zhao, Yang Zhou, Lei Li, Dimitris N. Metaxas

llms benchmarks

arXiv AI

4d ago

Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning

arXiv:2609.36759v1 Announce Type: cross Abstract: Benefiting from transferable visual-textual alignment, CLIP has been widely adopted for class-incremental learning (CIL). However, existing learners...

By Chiyuan He, Zihuan Qiu, Fanman Meng, Chao Wang, Liangjiang Chen, Linfeng Xu, Qingbo Wu, Hongliang Li

rag multimodal benchmarks safety

arXiv AI

Jun 4

EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation

arXiv:2603. 09493v2 Announce Type: replace-cross Abstract: The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge.

By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li

llms rag fine-tuning multimodal benchmarks

arXiv Computer Vision

Sep 1

Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs

arXiv:2507.00754v3 Announce Type: replace Abstract: The integration of Large Language Model (LLMs) blocks with Vision Transformers (ViTs) holds immense promise for vision-only tasks by leveraging the...

By Selim Kuzucu, Muhammad Ferjad Naeem, Anna Kukleva, Federico Tombari, Bernt Schiele

llms fine-tuning

arXiv Machine Learning

Jun 16

Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization

arXiv:2507. 02288v2 Announce Type: replace-cross Abstract: Domain Generalization (DG) seeks to develop a versatile model capable of performing effectively on unseen target domains.

By De Cheng, Zhipeng Xu, Xinyang Jiang, Dongsheng Li, Nannan Wang, Xinbo Gao

llms benchmarks safety

arXiv AI

Jul 31

Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

arXiv:2607. 26596v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture.

By Mingkuan Feng, Zhengqi Wen, Jianhua Tao

llms fine-tuning multimodal benchmarks

Related stories