arXiv Computer Vision
TIE: Training-free CLIP Embedding Guidance for Compositional Text-to-Image Alignment
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computer Vision
6d ago
ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
ProCLIP introduces a curriculum‑learning framework that progressively aligns a CLIP image encoder with an LLM‑based text embedder. It first distills CLIP’s text encoder knowledge into the LLM embedder, then fine‑tunes the image encoder via contrastive learning with self‑distillation, instance semantic alignment, and embedding structure alignment losses. The approach addresses CLIP’s token limit, lack of multilingual support, and preserves pre‑trained vision‑language alignment.
arXiv Computation and Language
Aug 21
Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment
arXiv:2608. 19825v1 Announce Type: cross Abstract: Medical image captioning is a technique that accelerates early-stage diagnostic workflows and enhances the interpretability of medical diagnostic AI systems.
arXiv Computer Vision
Oct 5
DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.
arXiv Computer Vision
Oct 16
Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching
Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.
arXiv Machine Learning
Oct 19
Domain-specific language alignment of multimodal embeddings towards explainable predictions
The paper introduces a framework that aligns multimodal embeddings with language by using captioning functions on interpretable auxiliary data, a soft contrastive loss, and a rank-based concept retrieval task. Applied to Earth observation embeddings, the method improves species distribution modelling, achieving state‑of‑the‑art performance on most metrics and better concept retrieval than prior vision‑language models. The aligned concepts enable recovery of species‑habitat associations and allow domain experts to interpret model predictions in natural language.
arXiv AI
Jul 28
Controlling Embedding Spaces with Text-Conditioned Transformations
arXiv:2607. 22919v1 Announce Type: cross Abstract: Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification.