arXiv Machine Learning September 10, 2026 By Siru Jiang, Yuwei Liang, Jian Liang, Ran He, Tieniu Tan

To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models

Read the original on arXiv Machine Learning →

The paper introduces selective adaptation for vision‑language models, questioning whether test‑time adaptation (TTA) should always be applied. By analyzing per‑sample predictions before and after adaptation, the authors find that many adaptations are negligible or even harmful, flipping correct predictions. They propose Cross‑Augmentation Similarity (CAS), which skips adaptation when predictions across augmented views are highly similar, achieving comparable or better accuracy while reducing adaptation by up to 85%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 15

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective

arXiv:2606. 14299v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment.

By Jiazhen Huang, Xiao Chen, Zhiming Liu, Yaru Sun, Jingyan Jiang, Zhi Wang

llms multimodal benchmarks

arXiv AI

Aug 28

Subspace Alignment for Vision-Language Model Test-time Adaptation

The paper introduces SubTTA, a test-time adaptation method for vision‑language models that aligns the semantic subspaces of visual and textual modalities to improve zero‑shot predictions. It addresses two issues: the modality gap caused by distribution shifts and visual nuisance that masks task‑specific semantics. By minimizing chordal distance between principal subspaces and projecting visual features onto a task‑specific textual subspace, SubTTA refines decision boundaries and achieves an average 2.24% improvement over existing TTA methods.

By Zhichen Zeng, Wenxuan Bao, Xiao Lin, Ruizhong Qiu, Tianxin Wei, Xuying Ning, Yuchen Yan, Chen Luo, Monica Xiao Cheng, Jingrui He, Hanghang Tong

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jul 13

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

arXiv:2607. 09450v1 Announce Type: cross Abstract: Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations.

By Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu, Jiannan Ge, Jiaheng Zhang, Long Chen

llms multimodal safety

arXiv Machine Learning

Jun 2

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.

By Yujun Li, Hongyuan Zhang, Yuan Yuan

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Aug 20

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

arXiv:2608. 18339v1 Announce Type: cross Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.

By Qi Yu, Zhichen Zeng, Katherine Tieu, Xiyuan Yang, Ruizhong Qiu, Yuchen Yan, Lihui Liu, Yanjun Zhao, Lingjie Chen, Jingrui He, Hanghang Tong

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 8

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

arXiv:2606. 06943v1 Announce Type: cross Abstract: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.

By Sunoh Kim, Daeho Um

llms efficiency multimodal benchmarks safety

One email a morning, machine-written

Related stories

What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective

Subspace Alignment for Vision-Language Model Test-time Adaptation

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models