← Back to all news
arXiv Computer Vision By Emanuel Ben Baruch, Sivan Biham, Alon Shoshan, Yochai Zvik, Avi Ben-Cohen

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

Read the original on arXiv Computer Vision →

The paper investigates how finetuning vision‑language models (VLMs) on class‑imbalanced data can severely hurt recall for rare classes, sometimes below zero‑shot performance. It discovers that using a different descriptive prompt at inference—termed cross‑prompt inference—recovers rare‑class recall across diverse VLMs and datasets, though it may slightly affect frequent‑class recall. Analysis with Tuning Contribution shows that cross‑prompt inference reduces reliance on finetuning‑induced computation, shifting the model toward its pretrained knowledge, and that the benefit disappears when the finetuning contribution is minimized.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 12

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.

By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
llmsmultimodal
More like this →
arXiv AI
Aug 19

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

The paper introduces ADAPT, an adversarial disentangled prompt tuning framework designed to improve the robustness of vision‑language models. ADAPT employs a dual‑prompt strategy: a target prompt learns robust features while a set of decoy prompts capture pseudo‑robust, non‑generalizable shortcuts. By enforcing orthogonality between target and decoy prompts, the method mitigates robust generalization overfitting and provides a theoretical error bound for unseen classes, leading to significant empirical robustness gains.

By Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang
llmsragmultimodalsafety
More like this →