← Back to all news
arXiv Computer Vision By Ellis Brown, Jihan Yang, Shusheng Yang, Rob Fergus, Saining Xie

Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

Read the original on arXiv Computer Vision →

The paper argues that multimodal large language models can answer vision benchmarks using only linguistic cues, revealing non-visual shortcuts. It proposes a Test‑set Stress‑Test (TsT) that trains a text‑only model on the test set to expose exploitable patterns, and introduces Iterative Bias Pruning (IBP) to remove high‑bias questions. Applying these methods to VSI‑Bench, CV‑Bench, and MMMU shows significant gaps between blind and vision‑aware performance and demonstrates that debiased benchmarks can reduce blind scores while preserving vision scores.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 12

Auditing VLM Benchmarks from Existing Evaluation Runs

The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.

By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov
llmsmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 22

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning

The paper introduces a free, label‑free visual evidence signal that improves fine‑grained vision‑language reasoning. By selecting image crops that maximize the model’s answer distribution peak, the method locates answer‑bearing regions without training or annotations, boosting accuracy from 70 % to 85 %. The evidence gap also complements model confidence, enabling better correctness prediction and error flagging.

By Santi Ram Tiwari, Nihal Naik, Devbrat Pandey, Nishant Sinha
llmsfine-tuningefficiencymultimodal
More like this →