arXiv AI By Byeongju Woo, Zilin Wang, Byeonghyun Pak, Sangwoo Mo, Stella X. Yu

Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding

Read the original on arXiv AI →

The paper introduces CAFT, a vision‑language model that improves understanding of long, detail‑rich captions by aligning local text regions with image parts before forming a global representation. It uses a fine‑to‑coarse image encoder and a part‑to‑whole text encoder to discover localized semantics and compose them into a holistic image‑text embedding. Trained on 30 M image‑text pairs, CAFT outperforms existing methods on six long‑text retrieval benchmarks and scales well, learning fine‑grained representations without explicit region supervision.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 24

CFM: Language-aligned Concept Foundation Model for Vision

The paper introduces CFM, a language‑aligned concept foundation model for vision that generates fine‑grained, human‑interpretable concepts with spatial grounding. By pairing CFM with a strong semantic foundation model, it provides explanations for downstream tasks such as classification, segmentation, and captioning. The authors also analyze local co‑occurrence of concepts to define relationships, improving concept naming and yielding richer explanations while maintaining competitive performance.

By Kai Wittenmayer, Sukrut Rao, Amin Parchami-Araghi, Bernt Schiele, Jonas Fischer
arXiv Computer Vision
Oct 16

Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.

By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
Hugging Face Trending Papers
Jul 15

Fine-grained CLIP fine-tuning with self-annotated region alignment

Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme.