Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The paper introduces Auto-Comp, a fully automated, concept-driven pipeline that generates photorealistic compositional benchmarks for vision‑language models. Auto‑Comp creates paired Minimal and Contextual samples for each concept, enabling isolation of core binding abilities from visio‑linguistic complexity. Evaluations across 25 models reveal consistent failures in attribute and relational binding, with context helping relational tasks but hindering attribute tasks due to visual clutter.
arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.
The paper introduces TIGER, a training framework that addresses the modality reasoning‑access gap in multimodal large language models (MLLMs). By generating synthetic counterfactuals where models succeed on text but fail on rendered images, TIGER fine‑tunes MLLMs using Group Relative Policy Optimization, SFT, and DPO, improving performance on real‑world visual benchmarks such as MathVerse, EMMA, MMMU, ChartQAPro, and RoboSpatial‑Home. The approach demonstrates that visual reasoning can be enhanced without multimodal annotations, and it complements existing RLVR methods.
CompoSET is a new vision‑language benchmark that tests compositionality by presenting each image‑caption pair with a foil that differs by only a single controlled edit—such as color, count, spatial relation, or object state—while keeping the overall scene largely unchanged. The dataset contains 1,776 instances across 80 scenes, 16 edit types in four semantic categories, and captions at three verbosity levels, with an average of only 8% pixel change between base and foil images. Evaluation of 22 dual‑encoder VLMs shows that all models perform at or below chance on CompoSET, contrasting with their strong performance on prior benchmarks that conflate global image differences with compositional reasoning.
arXiv:2605. 14054v2 Announce Type: replace Abstract: Achieving robust perception-reasoning synergy is a central goal for advanced Vision-Language Models (VLMs).
CS-CLIP is a vision‑language model that improves compositional reasoning by using scene graphs to identify compositional elements and create structured negative examples through selective masking. The approach retains only the most contradictory negatives, encouraging the model to depend on compositional structure instead of surface cues. CS-CLIP achieves state‑of‑the‑art performance on compositional reasoning benchmarks while maintaining strong cross‑modal retrieval and downstream visual reasoning capabilities with fewer training samples.