The paper shows that controlled visual in‑context learning can help vision‑language models (VLMs) overcome errors caused by training priors. By providing demonstrations, the models focus more on visual evidence, yielding significant accuracy gains on counterfactual examples while leaving prior‑aligned performance largely unchanged. This effect is observed across five VLM families and five vision‑centric tasks, indicating that many VLMs possess latent visual grounding abilities that standard tests may miss.
By Kun Wang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Esin Tureci
arXiv:2511. 16107v3 Announce Type: replace-cross Abstract: Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training.
By Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu
Multimodal large language models (MLLMs) have made strong progress on visual question answering and image captioning, yet they still produce fluent claims about objects, attributes, or relations that...
SAVOR is a training framework for multimodal large language models that adds token and answer confidence to the output schema, optimises a Group Relative Policy Optimisation objective to penalise calibration error and poor abstention, and uses the learned confidence at inference to revisit visual evidence only when uncertain. Experiments on POPE, HallusionBench, AMBER, and MMHal-Bench with InternVL3-8B and Qwen3-VL-8B backbones show that SAVOR reduces hallucination while maintaining general capability on MME and MMBench, achieving lower Expected Calibration Error than DPO and decoding baselines.
By Zixiu Ding, Zilin Zhao, Yingjie He, Xinlang Kang, Guansu Wang, Wei Zhang
arXiv:2609.13228v1 Announce Type: new
Abstract: Vision Language Models (VLMs) should rely on visual evidence that directly determines the correct answer, but supervision for grounding visual reasonin...
By Marko Jojic, Zhaonan Li, Ben Zhou
The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.
By Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
arXiv:2609.22588v1 Announce Type: new
Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...
By Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie
EDCT-Bench is a benchmark that evaluates the faithfulness of Vision‑Language Models (VLMs) by using Explanation‑Driven Counterfactual Testing (EDCT). EDCT extracts visual concepts from a model’s natural language explanation, applies minimal verified edits to those concepts, and checks whether the model’s answer and explanation remain consistent with the edited image. The benchmark covers three domains—knowledge‑intensive VQA, safety‑critical driving, and 3D spatial reasoning—and reveals significant faithfulness gaps in current VLMs, while also showing that EDCT‑generated counterfactuals can improve training.
By Sihao Ding, Santosh Vasa, Aditi Ramadwar, Thomas Monninger
arXiv:2607. 16311v1 Announce Type: cross Abstract: Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.
By Jingyu Sun, Jiachen Tu, Yuyang Xue, Yaoxin Jiang, Guoyi Xu, Zhengtao Yao, Rui Qian, Yizheng Sun, Hongpeng Zhou, Jingyuan Sun, Yan Lin
arXiv:2606. 29984v1 Announce Type: new Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs).
By Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma
arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.
By Runyu Zhou, Qi Zhang, Qixun Wang, Yisen Wang
The paper introduces AVA, a training‑free framework that improves interleaved multimodal in‑context learning for frozen large vision‑language models. AVA decomposes the interleaved context into per‑sample threads, selects influential demonstrations while removing redundancy, and calibrates consistency using a weak sibling model’s confidence. Across five vision‑language benchmarks, AVA raises performance by up to 2.4 points, narrows the visual performance gap on ScienceQA‑IMG, and remains robust even with 30% noisy demonstrations, all while keeping the LVLM frozen and adding only modest latency.
By Yigit Yilmaz, Elena Petrova, Mehmet Kaya, Lucia Rossi, Amir Rahman