DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.
By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
arXiv:2606.29997v2 Announce Type: replace
Abstract: Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limi...
By Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano, Daichi Yashima, Komei Sugiura
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set.
Flow-CLIP introduces a method to improve compositional image-text matching by refining global image-text alignment with image-conditioned changes in token–patch evidence. It combines Global Compatibility Calibration (GCC) for sentence-level similarity, Image-Conditioned Evidence Refinement (ICER) to incorporate visual patch context, and Cross-Image Margin Ranking (CIMR) to enhance ranking margins when images support correct descriptions. Experiments demonstrate that Flow-CLIP outperforms pretrained CLIP and other global alignment approaches on binding accuracy and reduces reliance on language priors.
By Minghong Wei, Enguang Zhu, Pu Cao, Lu Yang, Qing Song
arXiv:2511. 01390v2 Announce Type: replace-cross Abstract: Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications.
By Xinyu Mao, Junsi Li, Haoji Zhang, Yu Liang, Ming Sun
arXiv:2609.09973v1 Announce Type: new
Abstract: Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generate...
By Zizhen Wang, Bo Feng, Zhengfeng Lai, Shiyu Li, Yang Lu, Meng Cao, Ping Huang, Xiaoming Simon Wang
The paper proposes Metric-based Loss Weighting to enhance visual grounding in multimodal machine translation. By increasing loss for tokens that benefit from image context—identified via the Point-wise Cross-mutual Information (PCXMI) metric and its Congruency-based variant—the method improves translation accuracy on the CoMMuTE dataset by over 7 percentage points. Experiments fine-tune three pretrained multimodal LLMs across three language directions, showing superior performance compared to standard fine-tuning while preserving overall translation quality.
By Pawe{\l} M\k{a}ka, Piotr Andruszkiewicz, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis
SVG-Score introduces a human‑aligned evaluation framework for text‑to‑SVG generation, addressing the shortcomings of existing image‑based metrics like CLIPScore that poorly capture SVG‑specific errors such as color, count, and spatial inaccuracies. The authors first demonstrate that CLIP‑based scores are largely insensitive to these errors and that generic Vision‑Language Models respond inconsistently across error types and styles. They then present a human‑annotated Semantic Alignment dataset and develop two complementary evaluators: a CLIP‑based scorer adapted to vector graphics and a VLM judge refined through supervised fine‑tuning and reward‑shaped reinforcement learning, enabling both fast large‑scale and expressive, interpretable assessment of SVG generators.
By Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein, Afsana Mimi, Marcella Cornia, Lorenzo Baraldi, Gerard de Melo
arXiv:2603. 01696v2 Announce Type: replace-cross Abstract: Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions.
By Haonan Jia, Shichao Dong, Xin Dong, Zenghui Sun, Jin Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
TokenSwap introduces a method to benchmark and reduce the modality gap in multimodal large language models (MLLMs) by interleaving semantically aligned images with text tokens. By transforming text-only benchmarks like MMLU into image‑interleaved versions (TokenSwap‑Bench), the authors reveal a pervasive performance drop—averaging 19.6%—across 42 MLLMs, with reasoning models showing smaller gaps. Training with TokenSwap mitigates this gap while maintaining strong text‑only and vision‑language performance.
By Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.
By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim
arXiv:2607. 18237v1 Announce Type: cross Abstract: Human visual similarity judgments are context-dependent.
By Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, Alexei A. Efros, Richard Zhang