arXiv Computer Vision

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

The paper investigates how finetuning vision‑language models (VLMs) on class‑imbalanced data can severely hurt recall for rare classes, sometimes below zero‑shot performance. It discovers that using a different descriptive prompt at inference—termed cross‑prompt inference—recovers rare‑class recall across diverse VLMs and datasets, though it may slightly affect frequent‑class recall. Analysis with Tuning Contribution shows that cross‑prompt inference reduces reliance on finetuning‑induced computation, shifting the model toward its pretrained knowledge, and that the benefit disappears when the finetuning contribution is minimized.

arXiv Computer Vision

Oct 12

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.

By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci

llms multimodal

arXiv AI

Sep 30

Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

arXiv:2604.03114v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy...

By Zhangyun Tan, Zeliang Zhang, Jiani Liu, Susan Liang, Yolo Y. Tang, Lisha Chen, Chenliang Xu

llms multimodal safety

arXiv Machine Learning

Aug 11

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

arXiv:2605. 12705v2 Announce Type: replace Abstract: How can we train models whose post-trained capabilities survive subsequent fine-tuning?

By Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer, Ziqian Zhong, Aditi Raghunathan

fine-tuning

arXiv AI

Aug 5

In-Context Collapse in Vision-Language Models and How to Mitigate it?

arXiv:2608. 02830v1 Announce Type: cross Abstract: Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied.

By Mohammad Rostami

llms computer-vision multimodal benchmarks

arXiv AI

Aug 19

Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

The paper introduces ADAPT, an adversarial disentangled prompt tuning framework designed to improve the robustness of vision‑language models. ADAPT employs a dual‑prompt strategy: a target prompt learns robust features while a set of decoy prompts capture pseudo‑robust, non‑generalizable shortcuts. By enforcing orthogonality between target and decoy prompts, the method mitigates robust generalization overfitting and provides a theoretical error bound for unseen classes, leading to significant empirical robustness gains.

By Yang Chen, Zhan Zhuang, Yanbin Wei, Zebin Chen, Hua Liu, Yu Zhang

llms rag multimodal safety

arXiv AI

Jul 13

GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs

arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.

By Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

llms fine-tuning multimodal safety

arXiv AI

Jun 16

Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

arXiv:2603. 24058v2 Announce Type: replace-cross Abstract: Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis.

By Han Sun, Qin Li, Peixin Wang, Min Zhang

llms agents robotics multimodal benchmarks safety

arXiv AI

Oct 7

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

arXiv:2410.21582v4 Announce Type: replace-cross Abstract: Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of mainta...

By Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy, Pulkit Agrawal, Ila Fiete

computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jun 11

Mechanisms of Introspective Awareness

arXiv:2603. 21396v5 Announce Type: replace Abstract: Recent work has shown that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept -- a phenomenon termed "introspective awareness.

By Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, Jack Lindsey

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Computer Vision

Sep 22

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning

The paper introduces a free, label‑free visual evidence signal that improves fine‑grained vision‑language reasoning. By selecting image crops that maximize the model’s answer distribution peak, the method locates answer‑bearing regions without training or annotations, boosting accuracy from 70 % to 85 %. The evidence gap also complements model confidence, enabling better correctness prediction and error flagging.

By Santi Ram Tiwari, Nihal Naik, Devbrat Pandey, Nishant Sinha

llms fine-tuning efficiency multimodal

arXiv AI

5d ago

Prompt-agnostic Language Models

The paper introduces Prompt-agnostic Language Models, focusing on train‑time techniques to embed prompt robustness during instruction tuning. It shows that mixing paraphrases in training causes significant gradient interference—57% of per‑paraphrasing gradients conflict—leading to the development of two gradient‑merging methods, Prompt‑TIES and Prompt‑DARE, which resolve this interference. Experiments across four models demonstrate that these methods substantially reduce per‑example variance and improve local robustness with minimal impact on overall accuracy.

By Frederic Sadrieh

llms fine-tuning safety

arXiv Computer Vision

Sep 17

Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models

arXiv:2609.17790v1 Announce Type: new Abstract: Pre-trained vision-language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustnes...

By Akanksha Singh, Vinod K. Kurmi

llms multimodal

Related stories