arXiv Computation and Language September 2, 2026 By Shota Sato, Hajime Kiyama, Tosho Hirasawa, Mamoru Komachi

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

Read the original on arXiv Computation and Language →

The paper investigates why reducing the modality gap between image and text representations in CLIP does not always improve zero‑shot classification accuracy. It shows that while average alignment improves, the relative decision margins among classes can shift, leading to a prediction‑level hubness where predictions concentrate on a few classes. Experiments across datasets confirm that accuracy drops correlate with increased prediction concentration for both linear and learning‑based gap corrections.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

1d ago

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

The paper investigates how the modality gap— the separation between image and text representations in contrastive vision‑language models—affects different downstream tasks. By showing that a single dominant direction accounts for most of the image‑text mean separation, the authors explain why reducing or removing this gap can improve zero‑shot classification, degrade retrieval, or restore performance depending on the task. The study provides a geometric framework that clarifies when and why gap interventions should be applied in vision‑language systems.

By Aditya Sharma, Divya Saxena

llms rag multimodal safety

arXiv AI

Sep 10

Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

arXiv:2609.05730v1 Announce Type: cross Abstract: Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation...

By Samir Char, Carles Domingo-Enrich, Randall Balestriero

rag multimodal safety

arXiv Machine Learning

Jun 2

Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

arXiv:2606. 01710v1 Announce Type: cross Abstract: Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification.

By Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani

llms rag fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 28

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

arXiv:2501. 19060v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes.

By Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo

llms fine-tuning multimodal safety

arXiv AI

Aug 18

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

arXiv:2405. 17678v2 Announce Type: replace-cross Abstract: Achieving zero-shot adversarial robustness without sacrificing generalization remains challenging for foundation models such as CLIP, especially under large adversarial perturbations.

By Fengji Ma, Hei Victor Cheng, Chenxing Li, Li Liu

rag safety

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

One email a morning, machine-written

Related stories

One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models

Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models