← Back to all news
arXiv AI By Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

Read the original on arXiv AI →

The paper introduces HN-CLIP, a new objective for dense-caption retrieval that adapts similarity margins per negative example using the text encoder’s own geometry. By adding a detached caption‑similarity matrix to the negative logits, HN‑CLIP addresses the issue of near‑duplicate captions that cause premature loss saturation in InfoNCE training. Experiments on four benchmarks show that HN‑CLIP outperforms leading methods by 2.4–4.3 R@1, trains 2.4× faster than GOAL and 5.4× faster than StructXLIP, and achieves the best full‑data baseline with only 20% of the training data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 21

Half-Truths Break Similarity-Based Retrieval

The paper investigates how CLIP-style dual encoders sometimes increase similarity scores when a text description is extended with a plausible but incorrect detail, a phenomenon termed "half-truths." On the COCO dataset, CLIP correctly prefers the shorter, accurate description only 40.6% of the time, dropping to 32.9% when the added detail is a relation. The authors attribute this to weak supervision on caption components and propose CS-CLIP, a component-supervised fine-tuning approach that raises half-truth accuracy to 69.3% and improves compositional benchmark performance by 5.7 points.

By Bora Kargi, Arnas Uselis, Seong Joon Oh
multimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 5

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.

By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
llmsdiffusioncomputer-visionfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 16

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.

By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
multimodalbenchmarkssafety
More like this →