arXiv Computer Vision By Jiayi Lei, Yuandong Pu, Xingyu Han, Rongpeng Zhu, Jing Xu, Jinyao Wang, Zijian Zhou, Bin Fu, Yuewen Cao, Yihao Liu, Hongsheng Li

Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning

Read the original on arXiv Computer Vision →

The paper introduces CF-World, a counterfactual benchmark for evaluating causal reasoning in text-to-image models. CF-World presents scenarios at three levels—factual, explicit counterfactual, and implicit counterfactual—to test whether models can generate images that contradict real-world priors. Using CF-Eval, the authors find that all evaluated models perform worse on counterfactual prompts, highlighting challenges in following altered rules and recombining rare visual attributes.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI
Jul 21

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.

By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin
arXiv AI
Aug 10

Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

arXiv:2608. 06938v1 Announce Type: cross Abstract: The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions.

By Chen Ling, Hanqian Li, Dongnan Liu, Keyu Qian, Jungang Li, Xinglong liu, Shiyi Wang, Xin Dong, Pengcheng Zhu, Wei Zhou, Linjian Mo, Nai Ding
arXiv AI
Sep 10

Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models

The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.

By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu
arXiv Computer Vision
1d ago

InfiHall-Bench: A Benchmark for Hallucination Evaluation in Autoregressive Text-to-Image Generation

InfiHall-Bench is a new benchmark for evaluating hallucinations in visual autoregressive (VAR) text-to-image models, comprising 411 prompts that systematically combine five core visual skills—attribute binding, spatial relation, action relation, part consistency, and scene consistency. The benchmark reveals that VAR models exhibit persistent hallucinations across different autoregressive scales, prompting the authors to propose Confidence-Masked Hallucination Mitigation (CMHM), which focuses training on uncertain predictions and outperforms standard LoRA fine-tuning. InfiHall-Bench also demonstrates greater diagnostic power than existing diffusion-focused benchmarks by distinguishing models that appear comparable under current compositional tests.

By Yuxia Qiao, Xinhao Zhong, Junhao Li, Yi Sun, Jiaxin Hong, Shuoyang Sun, Bin Chen