The paper investigates how the modality gap— the separation between image and text representations in contrastive vision‑language models—affects different downstream tasks. By showing that a single dominant direction accounts for most of the image‑text mean separation, the authors explain why reducing or removing this gap can improve zero‑shot classification, degrade retrieval, or restore performance depending on the task. The study provides a geometric framework that clarifies when and why gap interventions should be applied in vision‑language systems.
By Aditya Sharma, Divya Saxena
arXiv:2609.05730v1 Announce Type: cross
Abstract: Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation...
By Samir Char, Carles Domingo-Enrich, Randall Balestriero
arXiv:2606. 01710v1 Announce Type: cross Abstract: Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification.
By Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani
arXiv:2501. 19060v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes.
By Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo
arXiv:2603.24528v2 Announce Type: replace
Abstract: Vision-language models like CLIP are trained with the objective of aligning text and image pairs. Beyond text prompts alone, recent works show that...
By Dipam Goswami, Simone Magistri, Gido M. van de Ven, Bart{\l}omiej Twardowski, Andrew D. Bagdanov, Tinne Tuytelaars, Joost van de Weijer
arXiv:2405. 17678v2 Announce Type: replace-cross Abstract: Achieving zero-shot adversarial robustness without sacrificing generalization remains challenging for foundation models such as CLIP, especially under large adversarial perturbations.
By Fengji Ma, Hei Victor Cheng, Chenxing Li, Li Liu
arXiv:2610.32407v1 Announce Type: new
Abstract: Text-to-image (T2I) diffusion models have advanced notably, but still fail to satisfy prompt conditions, resulting in attribute mismatches and missing...
By Sujung Hong, Tae Eun Choi, Youngjun Jun, Chanyong Yoon, Seong Jae Hwang
arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.
By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan
Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.
By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
The paper evaluates how vision-language models (VLMs) perform in image classification tasks, comparing language-guided and vision-only approaches across various dual-encoder VLMs such as SigLIP 2 and RADIOv2.5 on ImageNet-1k and its label-corrected variant. It identifies key factors influencing accuracy—including prompt design, class diversity, k‑NN neighbors, and reference set size—and demonstrates that language and vision provide complementary strengths. A simple, learning‑free fusion method based on per‑class precision is proposed to further improve classification performance.
By Illia Volkov, Nikita Kisel, Klara Janouskova, Jiri Matas
The paper introduces SubTTA, a test-time adaptation method for vision‑language models that aligns the semantic subspaces of visual and textual modalities to improve zero‑shot predictions. It addresses two issues: the modality gap caused by distribution shifts and visual nuisance that masks task‑specific semantics. By minimizing chordal distance between principal subspaces and projecting visual features onto a task‑specific textual subspace, SubTTA refines decision boundaries and achieves an average 2.24% improvement over existing TTA methods.
By Zhichen Zeng, Wenxuan Bao, Xiao Lin, Ruizhong Qiu, Tianxin Wei, Xuying Ning, Yuchen Yan, Chen Luo, Monica Xiao Cheng, Jingrui He, Hanghang Tong
arXiv:2609.10224v1 Announce Type: new
Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing acc...
By Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie