arXiv Machine Learning
Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computer Vision
Oct 12
Auditing VLM Benchmarks from Existing Evaluation Runs
The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.
arXiv AI
Aug 20
ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models
ReWEIGH the Evidence is a training‑free decoding technique that calibrates token‑level ordinal visual evidence to reduce hallucinations in large vision‑language models. It aggregates vocabulary ranks across visual positions, compares candidates to a token‑specific reference derived from unlabeled images, and applies a bounded penalty only when evidence falls below this reference. Experiments on four 7B backbones show up to a 21.3% reduction in hallucinated object mentions while largely preserving or improving descriptive and general performance, with minimal added latency.
Hugging Face Trending Papers
Aug 17
Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models
Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the...
arXiv Machine Learning
Sep 2
The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
arXiv:2609.00868v1 Announce Type: cross Abstract: Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual inp...
arXiv AI
Sep 10
Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.
arXiv Computer Vision
Sep 23
RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Models
arXiv:2609.26093v1 Announce Type: new Abstract: Vision-language models can answer spatial relation questions confidently even when the image supports an incompatible relation. We formulate relation-g...