← Back to all news
arXiv Computation and Language By Pengfeng Song, Shaozhong Zhang, Pengfei Shao

Hierarchical Fine-Grained Cross-Modal Alignment via Dynamic Granularity Discovery and Optimal Transport

Read the original on arXiv Computation and Language →

The paper introduces HFCMA, a hierarchical fine-grained cross‑modal alignment framework that transforms low‑level image patches and text tokens into adaptive visual regions and textual phrases through dynamic granularity discovery. It then performs structured region‑phrase matching using an entropy‑regularized optimal transport formulation, incorporating background debiasing, textual importance weighting, and prototype consistency regularization. Experiments on Flickr30K and MS‑COCO demonstrate that this approach improves fine‑grained image‑text retrieval by providing structured local alignment information and better discrimination of subtle semantic differences.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Oct 16

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.

By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
multimodalbenchmarkssafety
More like this →
arXiv Computer Vision
6d ago

Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking

Efficient Remote Sensing Image-Text Retrieval via Interaction-Free Recall and Text-Guided Reranking proposes a fast-then-fine (FTF) retrieval framework that first performs interaction-free multi-granular recall to quickly match query embeddings with pre-computed gallery embeddings, then refines the top‑K candidates using a parameter‑free text‑guided cross‑modal reranking module called the Balanced Text‑guided Interaction Block (BTIB). The method introduces an inter‑ and intra‑modal loss to jointly optimize cross‑modal alignment across multi‑granular representations. Experiments on public benchmarks show that FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.

By Xi Chen, Xu Chen, Xiamgyang Jia, Wei Wang, Xu Zhang, Zhenyuan Shun
ragmultimodalbenchmarkssafety
More like this →