← Back to all news
arXiv Machine Learning By Bora Kargi, Arnas Uselis, Seong Joon Oh

Half-Truths Break Similarity-Based Retrieval

Read the original on arXiv Machine Learning →

The paper investigates how CLIP-style dual encoders sometimes increase similarity scores when a text description is extended with a plausible but incorrect detail, a phenomenon termed "half-truths." On the COCO dataset, CLIP correctly prefers the shorter, accurate description only 40.6% of the time, dropping to 32.9% when the added detail is a relation. The authors attribute this to weak supervision on caption components and propose CS-CLIP, a component-supervised fine-tuning approach that raises half-truth accuracy to 69.3% and improves compositional benchmark performance by 5.7 points.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computer Vision
Oct 16

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.

By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
multimodalbenchmarkssafety
More like this →
arXiv AI
Aug 20

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.

By Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang
llmscomputer-visionfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 28

Preserve-and-Compose Training for Composed Image Retrieval

The paper introduces Preserve-and-Compose Training (PACT) for composed image retrieval, a task where a query image is modified by a textual instruction while preserving visual content from a reference image. PACT learns from image–text–text triplets, using target captions for supervision and visual evidence from the source image to maintain relevant details, without requiring target images or gallery updates. The authors also propose Chord scoring, which blends target similarity with source-relative directional agreement in a frozen image space, and demonstrate that this combined approach yields strong retrieval performance across multiple zero-shot CIR benchmarks and various backbones.

By Sehyun Kwon
benchmarks
More like this →
arXiv Machine Learning
Jul 28

Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

arXiv:2607. 23052v1 Announce Type: cross Abstract: Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like "umbrella and no person" retrieve images containing both, even when concept detection is reliable.

By Sultan Alshehri, Zhantao Yang, Han Zhang, Marios Savvides
llmsragfine-tuningmultimodal
More like this →