The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.
By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
ReWEIGH the Evidence is a training‑free decoding technique that calibrates token‑level ordinal visual evidence to reduce hallucinations in large vision‑language models. It aggregates vocabulary ranks across visual positions, compares candidates to a token‑specific reference derived from unlabeled images, and applies a bounded penalty only when evidence falls below this reference. Experiments on four 7B backbones show up to a 21.3% reduction in hallucinated object mentions while largely preserving or improving descriptive and general performance, with minimal added latency.
By Jihae Jeong, Junha Choi, Hwanjo Yu
Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the...
arXiv:2609.00868v1 Announce Type: cross
Abstract: Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual inp...
By Genpei Zhang
The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.
By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu
arXiv:2609.26093v1 Announce Type: new
Abstract: Vision-language models can answer spatial relation questions confidently even when the image supports an incompatible relation. We formulate relation-g...
By Feixiang Liu, Qiang Qiu, Qingyang Li, Hui Xu
arXiv:2608. 16805v1 Announce Type: cross Abstract: Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance.
By Yuanzhi Xu, Qian Gao, Jun Fan, Guohui Ding, Zhenyu Yang, Yuteng Xiao, Sixue Lin
arXiv:2610.23610v1 Announce Type: cross
Abstract: Object hallucination in vision-language models (VLMs) is usually measured by checking each object a caption names against the image. Most benchmarks...
By Hsin-Ling Hsu, Neeraja Kirtane, Kuan-Hao Huang
arXiv:2608.21832v1 Announce Type: new
Abstract: Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether mo...
By Md Abrar Jahin, Md Rizwan Parvez
arXiv:2609.13308v1 Announce Type: cross
Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language...
By Sarthak Sattigeri
The paper investigates how different design choices in visual-head discovery scores affect the attention heads selected in vision‑language models. By systematically varying query type, discovery corpus, and attention source in Qwen3‑VL‑8B‑Instruct, the authors find that the attention source and query type have a larger impact on head selection than the corpus. While necessity scores distinguish discovery settings, steerability does not show a significant advantage over random head sets.
By Md Abdul Kadir, Daniel Sonntag
The paper argues that apparent capability limits in vision‑language benchmarks often stem from the way answers are presented rather than from the models themselves. By comparing performance on COCO images with answer choices given as English names versus pixel coordinates, the authors show that models like Qwen3‑VL‑4B perform far better when answers are in natural language, and that the choice of answer format can swing model rankings by dozens of points. The study also demonstrates that different conventions (e.g., hue angles vs. pixel coordinates) reveal which formats a model can actually interpret, highlighting that a fixed answer vocabulary is not neutral across models.
By Alfredo F. Frontera Del Valle