The paper introduces selective adaptation for vision‑language models, questioning whether test‑time adaptation (TTA) should always be applied. By analyzing per‑sample predictions before and after adaptation, the authors find that many adaptations are negligible or even harmful, flipping correct predictions. They propose Cross‑Augmentation Similarity (CAS), which skips adaptation when predictions across augmented views are highly similar, achieving comparable or better accuracy while reducing adaptation by up to 85%.
arXiv:2606. 14299v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment.
The paper introduces SubTTA, a test-time adaptation method for vision‑language models that aligns the semantic subspaces of visual and textual modalities to improve zero‑shot predictions. It addresses two issues: the modality gap caused by distribution shifts and visual nuisance that masks task‑specific semantics. By minimizing chordal distance between principal subspaces and projecting visual features onto a task‑specific textual subspace, SubTTA refines decision boundaries and achieves an average 2.24% improvement over existing TTA methods.
arXiv:2607. 09450v1 Announce Type: cross Abstract: Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations.
arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.
arXiv:2608. 18339v1 Announce Type: cross Abstract: Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.
arXiv:2606. 06943v1 Announce Type: cross Abstract: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.
arXiv:2607. 03900v1 Announce Type: cross Abstract: Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.
The paper introduces a method for selecting the best vision‑language model for a downstream task by analyzing the internal dynamics of the visual encoder. It represents each task with layer‑wise conductance and uses an entropy‑regularized alignment to derive a target‑conditioned block importance distribution. The proposed Directional Conductance Divergence (DCD) metric captures asymmetric transferability, enabling accurate prediction of model rankings without direct inference, and achieves a 14.7% NDCG@5 improvement over SWAB on 48 VLMs across 21 datasets.
The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
The paper introduces Test-Time Logit Prompting (TLP), a lightweight framework that adapts vision-language models to missing-modality inputs without accessing source training data. TLP optimizes logit prompts using uncertainty-aware adjustments and modality-complete consistency regularization, thereby maintaining prediction confidence and semantic consistency. Experiments on various benchmarks show that TLP improves recognition performance by up to 8% while requiring only a few hundred tunable parameters and minimal test-time optimization steps.
arXiv:2608.29395v1 Announce Type: new Abstract: Vision-language models such as CLIP and SigLIP provide strong zero-shot recognition, but their predictions can degrade when deployed on target data tha...