arXiv AI

Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models

The paper introduces Auto-Comp, a fully automated, concept-driven pipeline that generates photorealistic compositional benchmarks for vision‑language models. Auto‑Comp creates paired Minimal and Contextual samples for each concept, enabling isolation of core binding abilities from visio‑linguistic complexity. Evaluations across 25 models reveal consistent failures in attribute and relational binding, with context helping relational tasks but hindering attribute tasks due to visual clutter.

arXiv Computer Vision
Sep 28

Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis

arXiv:2609.31456v1 Announce Type: new Abstract: Vision-language models (VLMs) often struggle with compositional reasoning tasks, but the reasons for this underperformance remain unclear. A common hyp...

By Mona Gandhi, Cenk Merih Olcay, Kuan-Chieh Lo, Santiago Castro, Christopher W. Myers, Srinivasan Parthasarathy
arXiv Computer Vision
Nov 2

CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

CompoSET is a new vision‑language benchmark that tests compositionality by presenting each image‑caption pair with a foil that differs by only a single controlled edit—such as color, count, spatial relation, or object state—while keeping the overall scene largely unchanged. The dataset contains 1,776 instances across 80 scenes, 16 edit types in four semantic categories, and captions at three verbosity levels, with an average of only 8% pixel change between base and foil images. Evaluation of 22 dual‑encoder VLMs shows that all models perform at or below chance on CompoSET, contrasting with their strong performance on prior benchmarks that conflate global image differences with compositional reasoning.

By Laure Delisle, Pietro Perona
arXiv AI
6d ago

Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER

The paper introduces TIGER, a training framework that addresses the modality reasoning‑access gap in multimodal large language models (MLLMs). By generating synthetic counterfactuals where models succeed on text but fail on rendered images, TIGER fine‑tunes MLLMs using Group Relative Policy Optimization, SFT, and DPO, improving performance on real‑world visual benchmarks such as MathVerse, EMMA, MMMU, ChartQAPro, and RoboSpatial‑Home. The approach demonstrates that visual reasoning can be enhanced without multimodal annotations, and it complements existing RLVR methods.

By Gregory Kang Ruey Lau, Minh Huynh Nguyen, Bryan Kian Hsiang Low
arXiv Machine Learning
Oct 9

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

WOVEN is a new training source and benchmark for visual transition reasoning, comprising 36,076 examples across 20 scenes, 5 actions, and 8 reasoning types. The study shows that current multimodal large language models (MLLMs) have a systematic deficit in spatial, embodied, physical, and temporal reasoning, and that training on WOVEN improves performance on 22 of 26 external benchmarks by up to 27.3 percentage points. The authors also provide a training recipe that prioritizes reasoning operations and larger visual changes for robust visual world modeling.

By Zheyu Fan, Yue Zhang, Mingkai Deng, Kangrui Wang, Qineng Wang, Canyu Chen, Jie Hao, Xing Fan, Chenlei Guo, Eric P. Xing, Mohit Bansal, Manling Li
arXiv Computer Vision
Oct 19

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

The paper introduces SABER, a benchmark that tests attribute–object binding in vision‑language models across all four modality pairings and scenes with up to ten objects. Experiments show significant binding failures in both CLIP‑style dual encoders and recent MLLM‑based embedders. The authors propose a binding‑focused fine‑tuning strategy using synthetic attribute‑binding text, which improves SABER performance and also boosts results on negation and instance‑retrieval benchmarks.

By Netanel Y. Tamir, Michael Green, Issar Tzachor, Or Shimshi, Dvir Samuel, Or Patashnik, Rami Ben-Ari