arXiv AI

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.

arXiv Computer Vision

3d ago

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

arXiv:2609.10224v1 Announce Type: new Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing acc...

By Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie

llms rag multimodal safety

arXiv Machine Learning

Jun 30

Contrastive vision-language learning with paraphrasing and negation

arXiv:2511. 16527v2 Announce Type: replace-cross Abstract: Contrastive vision-language models continue to be the dominant approach for image-text retrieval.

By Kwun Ho Ngan, Saman Sadeghi Afgeh, Joe Townsend, Artur d'Avila Garcez

llms rag diffusion computer-vision multimodal

arXiv Computer Vision

Sep 2

A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

arXiv:2609.00591v1 Announce Type: new Abstract: An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level c...

By Suryaansh Jain, Rahasya Barkur, Vishal G, Ryan Rossi, Franck Dernoncourt, Jack Wang, Koustava Goswami, Nedim Lipka, Puneet Mathur, Samyadeep Basu, Seunghyun Yoon

llms rag fine-tuning efficiency multimodal

arXiv AI

Jun 8

TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

arXiv:2606. 07451v1 Announce Type: cross Abstract: Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space.

By Sweta Mahajan, Sukrut Rao, Jiahao Xie, Alexander Koller, Bernt Schiele

llms rag multimodal benchmarks safety

arXiv AI

Jun 26

Reconstruction Alignment Improves Unified Multimodal Models

arXiv:2509. 07295v4 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture.

By Ji Xie, Trevor Darrell, Luke Zettlemoyer, XuDong Wang

rag diffusion multimodal benchmarks safety

arXiv AI

Aug 3

Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

arXiv:2607. 28986v1 Announce Type: cross Abstract: Zero-shot image captioning (ZIC) describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers.

By Duy Tran Thanh, Thien-Phuc Doan, Long Nguyen-Vu, Ngo Tan Vu Khanh

llms rag agents efficiency multimodal safety

arXiv AI

3d ago

Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

arXiv:2609.05730v1 Announce Type: cross Abstract: Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation...

By Samir Char, Carles Domingo-Enrich, Randall Balestriero

rag multimodal safety

arXiv AI

Jun 8

Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets

arXiv:2606. 07032v1 Announce Type: cross Abstract: Zero-Shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image based on a query composed of a reference image and a relative caption without training samples.

By Zhenyu Yang, Zemin Du, Shengsheng Qian, Changsheng Xu

llms benchmarks

arXiv AI

Jul 3

SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

arXiv:2511. 01390v2 Announce Type: replace-cross Abstract: Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications.

By Xinyu Mao, Junsi Li, Haoji Zhang, Yu Liang, Ming Sun

llms diffusion nlp multimodal safety

arXiv AI

Jun 30

Multimodal Representation Alignment for Cross-modal Information Retrieval

arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.

By Fan Xu, Luis A. Leiva

llms rag multimodal benchmarks safety

arXiv AI

Sep 1

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

arXiv:2608.29313v1 Announce Type: cross Abstract: CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embed...

By Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat, Seifeldin Elkerdany, Weixing Wang, Gerard de Melo

llms rag multimodal safety

arXiv AI

Jul 28

What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

arXiv:2607. 23271v1 Announce Type: cross Abstract: Contrastive vision-language models such as CLIP map semantically opposite phrases (e.

By Chen-Yi Lu, Yueh-Shao Chen, Somali Chaterji

llms rag diffusion fine-tuning multimodal safety

Related stories