arXiv Computer Vision

On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

The paper investigates few-shot adaptation of vision-language models, noting that gradient magnitudes are poorly calibrated while the signs of a few top-magnitude gradients remain informative. It proposes two state‑free optimizers, SignSparseSGD and DynSignSparseSGD, which use gradient magnitudes only to select update locations and fixed‑magnitude signs to determine update directions. Across 11 datasets, three CLIP backbones, and five shot levels, these methods match or surpass parameter‑efficient and optimization‑based baselines, show reduced sensitivity to long training schedules, and require no task‑specific modules or optimizer state beyond a single shared configuration.

arXiv AI

5d ago

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

SD‑LoRA introduces a training‑time structural distillation framework that embeds token‑level graph reasoning into a lightweight LoRA‑adapted CLIP model for few‑shot vision‑language adaptation. During training, a heterogeneous graph teacher captures patch‑patch, global‑local, and visual‑text relations over high‑resolution ViT tokens, and its structural bias is transferred to the CLIP‑LoRA backbone via implicit distillation and Prototype Predictive Alignment. At inference, the graph teacher is discarded, requiring only a single CLIP‑LoRA forward pass and a cache‑based classifier, yielding significant accuracy gains across 11 few‑shot benchmarks without extra computation.

By Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar, Yonghuai Liu

llms rag fine-tuning efficiency multimodal benchmarks safety

Hugging Face Trending Papers

Jul 20

PRiSM: Prototype Regularization for Few-Shot VLMs

Training-free few-shot adaptation methods have gained significant attention recently in the context of Vision-language Models (VLMs). Yet, current benchmarks rely on strong assumptions about the statistics of the adaptation data, e.

llms multimodal benchmarks safety

arXiv AI

Oct 2

VETO: Video Efficient Token Optimization for Vision Language Models

VETO (Video Efficient Token Optimization for Vision Language Models) is a plug‑in that reduces the quadratic cost of visual tokens in long‑video inference by applying dual‑axis compression: an intra‑frame compressor merges semantically similar tokens within each frame, and an inter‑frame compressor merges temporally redundant frames. By first compressing spatial dimensions, VETO lowers the cost of subsequent global temporal matching, surpassing single‑axis methods and achieving up to 45% faster inference on models such as LLaVA‑OneVision‑7B while maintaining or improving accuracy. The approach is universally applicable across LLaVA‑OneVision, InternVL‑2.5, and LongVA, preserving or enhancing zero‑shot accuracy even under extreme token budgets.

By Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen, Winston H. Hsu

llms multimodal

arXiv Machine Learning

Aug 6

DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models

arXiv:2608. 04496v1 Announce Type: cross Abstract: Visual inputs in vision-language models (VLMs) are often encoded into substantially longer token sequences than text, making visual tokens a major bottleneck for efficient inference.

By Chen Zhong, Xiao An, Zijie Wang, Jiepan Li, Guangyi Yang, Wei He

llms efficiency multimodal benchmarks

arXiv AI

Sep 10

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

arXiv:2609.06967v1 Announce Type: cross Abstract: Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many exist...

By Seungmin Oh, Seunghun Kang, Jongbin Ryu

llms fine-tuning multimodal

arXiv AI

Jun 2

On the Limits of Token Reduction for Efficient Unified Vision Language Training

arXiv:2606. 01503v1 Announce Type: cross Abstract: Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective.

By Siyi Chen, Weiming Zhuang, Jingtao Li, Lingjuan Lv

llms multimodal

arXiv AI

Oct 21

Language Guidance for Supervised Vision Training: An Empirical Study of Generalization

The paper investigates whether pretrained language models can act as lightweight auxiliary supervision for supervised vision training, without requiring paired image-text data or complex objectives. Two forms of language guidance—Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG)—are evaluated across six generalization regimes, including in‑distribution, out‑of‑distribution, shortcut resistance, texture/shape bias, adversarial robustness, and continual learning. Results show that ExLG improves in‑distribution, low‑data performance, and continual learning retention, while ImLG is more effective in shortcut‑sensitive and adversarial settings, and both add minimal parameters and training overhead.

By Shruthi Gowda, Nikita Jain, Elahe Arani, Bahram Zonooz

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 31

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

arXiv:2607. 28627v1 Announce Type: cross Abstract: Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints.

By Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem

llms rag efficiency multimodal benchmarks

arXiv AI

Jun 8

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

arXiv:2606. 06943v1 Announce Type: cross Abstract: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.

By Sunoh Kim, Daeho Um

llms efficiency multimodal benchmarks safety

arXiv Computer Vision

Sep 1

GATE: Reliability-Gated Gaussian Evidence Fusion for Training-Free Test-Time Adaptation of Vision-Language Models

arXiv:2608.29395v1 Announce Type: new Abstract: Vision-language models such as CLIP and SigLIP provide strong zero-shot recognition, but their predictions can degrade when deployed on target data tha...

By Pedram MohajerAnsari, Amir Salarpour, Run Wang, Mert D. Pes\'e

llms multimodal benchmarks

arXiv Computer Vision

Oct 15

Enabling Validation for Robust Few-Shot Recognition

arXiv:2506.04713v4 Announce Type: replace Abstract: A key challenge in Few-Shot Recognition (FSR) is the scarcity of labeled task-specific data for training. Contemporary methods mitigate this challe...

By Hanxin Wang, Tian Liu, Shu Kong

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 10

On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data

arXiv:2609.10321v1 Announce Type: new Abstract: Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in curren...

By Hongyuan Zhang, Xianda Guo, Yanlun Peng, Qianlong Yang, Yubin Guo, Pinhan Fu, Mulin Chen, Xiaozhen Qiao, Ping Luo

llms efficiency multimodal benchmarks

Related stories