arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.
By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin
The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.
By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu
The paper investigates how the order of generating explanations—whether a rationale is produced before or after the answer—affects vision‑language reasoning. By conducting controlled experiments on knowledge‑intensive QA, visual entailment, and compositional grounding tasks, the authors show that larger models are required for reliable rationale‑first generation, while answer‑first generation is less susceptible to format errors. The study concludes that explanation ordering, model scale, pre‑training knowledge, fine‑tuning, and task structure jointly influence prediction accuracy and reasoning faithfulness.
By Siting Liang, Luca Rippe, Omar Adjali, Daniel Sonntag
arXiv:2608. 08021v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context.
By Haojie Huang, Xinlei Yu, Chengming Xu, Zhangquan Chen, Cheng Yang, Qingdong He, Yu Yang, Jiangning Zhang, Xiaobin Hu
arXiv:2609.13228v1 Announce Type: new
Abstract: Vision Language Models (VLMs) should rely on visual evidence that directly determines the correct answer, but supervision for grounding visual reasonin...
By Marko Jojic, Zhaonan Li, Ben Zhou
arXiv:2610.16977v1 Announce Type: cross
Abstract: Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depend...
By Shu Zhao, Tan Yu