VisionQ is a new benchmark for qualitative analysis in computer vision that evaluates vision‑language models (VLMs) on criterion‑conditioned visual discrimination. It is built from over 1,800 peer‑reviewed comparison figures in CVPR and ICCV papers, linking each image crop to author‑stated visual claims through 3,911 hand‑annotated data points. The benchmark includes a 51‑leaf taxonomy of visual criteria, a protocol that hides method identities and reports accuracy per criterion, and a DPO‑tuned Gemma‑4‑E4B judge that improves accuracy on a held‑out test set.
By Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao, Vu Quynh Giao, Quang Hong Nguyen, Binh-Son Hua, Barry O'Sullivan, David Murphy, Hoang D. Nguyen
arXiv:2609.09417v1 Announce Type: new
Abstract: Vision-language models (VLMs) show promise for agricultural classification, but zero-shot performance on disease, pest, damage, quality, and species id...
By Earl Ranario, Jared Smith, Lars Lundqvist, Urmil Jatin Chandarana
OccludeBench is a manually curated benchmark that tests vision‑language models (VLMs) on their ability to recover fully occluded objects from remaining visual evidence. The benchmark distinguishes direct cases—requiring recovery from a specific physical cue such as a shadow or reflection—from inferential cases, which rely on broader scene context. Across five VLMs, accuracy averages 42.7 % on direct cases versus 62.7 % on inferential cases, a 20‑point gap, while humans achieve 63.4 % and 68.5 % respectively with no significant gap. The study introduces OccludeTrace, a four‑stage diagnostic that isolates the failure point, revealing that models struggle most when recovering the relevant object from the full image projection. Interventions such as inpaint‑then‑confront reduce the gap to a statistically non‑significant difference for models with image‑generation capabilities.
By Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu
arXiv:2606. 08123v2 Announce Type: replace-cross Abstract: Model selection for safety-relevant visual recognition is often based on clean aggregate performance, although robustness, transfer, embedded latency, and explanation faithfulness may produce different preferences.
By Ruben Dario Florez-Zela
arXiv:2610.03389v1 Announce Type: cross
Abstract: Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every...
By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Shaoyi Huang, Pu Zhao, Yanzhi Wang
The paper demonstrates that a targeted adversarial perturbation can reduce a vision‑language model’s training loss to near zero for a fixed target caption, yet the same model, when generating freely, still produces the correct description. This phenomenon, termed the train/inference gap, is traced to a single autoregressive step where the target token’s rank is fixed across all images, and further analysis shows that the language decoder, rather than the visual encoder, determines whether the corrupted signal is amplified or suppressed. The study uses a controlled two‑stage PGD attack on Qwen2.5‑VL‑7B‑Instruct and evaluates the effect on 200 held‑out COCO images, revealing that adversarial robustness in autoregressive VLMs largely depends on the language decoder’s prior.
whyItMatters":"The findings suggest that defenses and faithfulness evaluations for deployed vision‑language models should focus on the language decoder rather than the visual encoder, as the former is the key determinant of robustness to adversarial perturbations."
By Arun Josephraj Arokiaraj, Zekun Wu, Adriano Koshiyama
arXiv:2608.23253v1 Announce Type: cross
Abstract: Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Exi...
By Taoyu Qian, Qi Wang, Daqian Shi, Yuanhao Jiang, Shang Gao, Hualong Yu
arXiv:2610.15855v1 Announce Type: new
Abstract: We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a larg...
By Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van Durme, Andrew Yates, Reno Kriz, William Walden
arXiv:2605.26380v2 Announce Type: replace-cross
Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...
By Jingru Chen, Yiming Liu, Mingtao Chen, Sijie Chen, Richeng Xuan, Liang Yang, Zhichao Hu, Fanyang Lu
arXiv:2607. 24354v1 Announce Type: new Abstract: Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results.
By Haoyue Liu, Xiaoyu Ma, Ye Chen, Yuexian Zou, Xiaoying Tang
arXiv:2610.22610v1 Announce Type: cross
Abstract: Vision-language models (VLMs) are usually chosen from leaderboards, which give one ranking for every application. On questions about objects that a s...
By Faisal Mohammad, Mohamed Younis
The paper investigates why multimodal large language models (MLLMs) struggle with vision‑centric tasks when visual evidence conflicts with pretrained language knowledge. Using image reconstruction and a new WhatIfVis benchmark, the authors show that MLLMs preserve coarse‑grained visual attributes but fail to consistently use them, and that supervised fine‑tuning and activation patching can improve controllability of visual context sensitivity. The study demonstrates that the main bottleneck lies in the models’ inability to reliably regulate their reliance on visual evidence rather than in visual perception itself.
By Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan, Xi Liu, Serge Belongie, V\'esteinn Sn{\ae}bjarnarson