← Back to all news
arXiv Computer Vision By Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini, Eric Granger

On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

Read the original on arXiv Computer Vision →

The paper investigates few-shot adaptation of vision-language models, noting that gradient magnitudes are poorly calibrated while the signs of a few top-magnitude gradients remain informative. It proposes two state‑free optimizers, SignSparseSGD and DynSignSparseSGD, which use gradient magnitudes only to select update locations and fixed‑magnitude signs to determine update directions. Across 11 datasets, three CLIP backbones, and five shot levels, these methods match or surpass parameter‑efficient and optimization‑based baselines, show reduced sensitivity to long training schedules, and require no task‑specific modules or optimizer state beyond a single shared configuration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
5d ago

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

SD‑LoRA introduces a training‑time structural distillation framework that embeds token‑level graph reasoning into a lightweight LoRA‑adapted CLIP model for few‑shot vision‑language adaptation. During training, a heterogeneous graph teacher captures patch‑patch, global‑local, and visual‑text relations over high‑resolution ViT tokens, and its structural bias is transferred to the CLIP‑LoRA backbone via implicit distillation and Prototype Predictive Alignment. At inference, the graph teacher is discarded, requiring only a single CLIP‑LoRA forward pass and a cache‑based classifier, yielding significant accuracy gains across 11 few‑shot benchmarks without extra computation.

By Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar, Yonghuai Liu
llmsragfine-tuningefficiencymultimodalbenchmarkssafety
More like this →
arXiv AI
Oct 2

VETO: Video Efficient Token Optimization for Vision Language Models

VETO (Video Efficient Token Optimization for Vision Language Models) is a plug‑in that reduces the quadratic cost of visual tokens in long‑video inference by applying dual‑axis compression: an intra‑frame compressor merges semantically similar tokens within each frame, and an inter‑frame compressor merges temporally redundant frames. By first compressing spatial dimensions, VETO lowers the cost of subsequent global temporal matching, surpassing single‑axis methods and achieving up to 45% faster inference on models such as LLaVA‑OneVision‑7B while maintaining or improving accuracy. The approach is universally applicable across LLaVA‑OneVision, InternVL‑2.5, and LongVA, preserving or enhancing zero‑shot accuracy even under extreme token budgets.

By Gueter Josmy Faure, Hao Ping Wang, Min-Hung Chen, Winston H. Hsu
llmsmultimodal
More like this →