arXiv Computer Vision

Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning

The paper introduces CF-World, a counterfactual benchmark for evaluating causal reasoning in text-to-image models. CF-World presents scenarios at three levels—factual, explicit counterfactual, and implicit counterfactual—to test whether models can generate images that contradict real-world priors. Using CF-Eval, the authors find that all evaluated models perform worse on counterfactual prompts, highlighting challenges in following altered rules and recombining rare visual attributes.

arXiv AI
Jul 21

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.

By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin
arXiv AI
Aug 10

Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

arXiv:2608. 06938v1 Announce Type: cross Abstract: The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions.

By Chen Ling, Hanqian Li, Dongnan Liu, Keyu Qian, Jungang Li, Xinglong liu, Shiyi Wang, Xin Dong, Pengcheng Zhu, Wei Zhou, Linjian Mo, Nai Ding
arXiv AI
Sep 10

Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models

The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.

By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu
arXiv Computer Vision
1d ago

InfiHall-Bench: A Benchmark for Hallucination Evaluation in Autoregressive Text-to-Image Generation

InfiHall-Bench is a new benchmark for evaluating hallucinations in visual autoregressive (VAR) text-to-image models, comprising 411 prompts that systematically combine five core visual skills—attribute binding, spatial relation, action relation, part consistency, and scene consistency. The benchmark reveals that VAR models exhibit persistent hallucinations across different autoregressive scales, prompting the authors to propose Confidence-Masked Hallucination Mitigation (CMHM), which focuses training on uncertain predictions and outperforms standard LoRA fine-tuning. InfiHall-Bench also demonstrates greater diagnostic power than existing diffusion-focused benchmarks by distinguishing models that appear comparable under current compositional tests.

By Yuxia Qiao, Xinhao Zhong, Junhao Li, Yi Sun, Jiaxin Hong, Shuoyang Sun, Bin Chen
arXiv AI
Sep 17

EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

EDCT-Bench is a benchmark that evaluates the faithfulness of Vision‑Language Models (VLMs) by using Explanation‑Driven Counterfactual Testing (EDCT). EDCT extracts visual concepts from a model’s natural language explanation, applies minimal verified edits to those concepts, and checks whether the model’s answer and explanation remain consistent with the edited image. The benchmark covers three domains—knowledge‑intensive VQA, safety‑critical driving, and 3D spatial reasoning—and reveals significant faithfulness gaps in current VLMs, while also showing that EDCT‑generated counterfactuals can improve training.

By Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger
arXiv Computer Vision
Oct 12

WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation

WorldGenBench is a benchmark that tests text‑to‑image models on their ability to combine factual grounding with contextual reasoning. It introduces the Knowledge Checklist Score to evaluate whether generated images reflect both explicit and implicit semantic elements. The study finds that while some proprietary models perform well, most open‑source systems struggle, especially with culturally diverse and scientific content, revealing common failure modes such as semantic hallucination and underrepresentation of non‑Western knowledge.

By Ruoshi Xu, Daoan Zhang, Che Jiang, Biaoxiang Chen, Zijian Jin, Yutian Lu, Jianguo Zhang, Liang Yong, Jiebo Luo, Shengda Luo
arXiv Computer Vision
Sep 28

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

CCRV-Bench is a constraint‑driven benchmark designed to evaluate visual causal reasoning in vision‑language models on single‑image physical scenarios. It assesses four causal task dimensions—causal relation discovery, state prediction, causal diagnosis, and intervention—while applying constraints such as entity symbolization, spatial grounding, factual adversarial constraints, and minimalist output constraints to reduce shortcut learning. Experiments on 15 multimodal models reveal that constraint sensitivity varies by task and model, with intervention and spatial grounding having the largest impact and factual adversarial constraints improving causal diagnosis across models.

By Linyuan Gao, Yuan Wu, Yi Chang
arXiv AI
Oct 21

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.

By Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey
arXiv Machine Learning
Nov 5

EPIC: Efficient Predicate-Guided Inference-Time Control for Compositional Text-to-Image Generation

EPIC is a training‑free inference‑time refinement framework that improves compositional text‑to‑image generation by parsing prompts into visual programs and verifying candidate images against typed predicates. It guides refinement through localized editing or resampling based on unmet predicates, achieving a Soft‑TIFA GM score of 71.46% on GenEval2 compared to 34.16% for single‑pass generation. EPIC also reduces image‑model executions by 31%, MLLM calls by 72%, and MLLM tokens by 81% while outperforming the strongest baseline by 19.23 points.

By Sunung Mun, Sunghyun Cho, Jungseul Ok