arXiv Computation and Language By Shota Sato, Hajime Kiyama, Tosho Hirasawa, Mamoru Komachi

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

Read the original on arXiv Computation and Language →

The paper investigates why reducing the modality gap between image and text representations in CLIP does not always improve zero‑shot classification accuracy. It shows that while average alignment improves, the relative decision margins among classes can shift, leading to a prediction‑level hubness where predictions concentrate on a few classes. Experiments across datasets confirm that accuracy drops correlate with increased prediction concentration for both linear and learning‑based gap corrections.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Sep 30

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

The paper investigates how the modality gap— the separation between image and text representations in contrastive vision‑language models—affects different downstream tasks. By showing that a single dominant direction accounts for most of the image‑text mean separation, the authors explain why reducing or removing this gap can improve zero‑shot classification, degrade retrieval, or restore performance depending on the task. The study provides a geometric framework that clarifies when and why gap interventions should be applied in vision‑language systems.

By Aditya Sharma, Divya Saxena
arXiv Computer Vision
Oct 1

Project and Mix: Task-Semantic Prototypes for Few-Shot Image Classification

arXiv:2603.24528v2 Announce Type: replace Abstract: Vision-language models like CLIP are trained with the objective of aligning text and image pairs. Beyond text prompts alone, recent works show that...

By Dipam Goswami, Simone Magistri, Gido M. van de Ven, Bart{\l}omiej Twardowski, Andrew D. Bagdanov, Tinne Tuytelaars, Joost van de Weijer