arXiv Machine Learning

More Accurate, Less Human: Gestalt Grouping in Vision Models

arXiv:2608. 10195v1 Announce Type: cross Abstract: Human vision organizes what it sees into wholes: same-colored points group into series, similar marks cohere into categories, and shapes complete into recognizable objects.

arXiv AI

Aug 20

When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't

The paper introduces the Graded Color Attribution (GCA) dataset, a benchmark that tests whether Vision‑Language Models (VLMs) and humans can articulate and follow a threshold rule for labeling objects by color. In experiments, humans consistently adhere to their stated rules, while VLMs—despite accurately estimating color coverage—often violate their own introspective rules, especially when world‑knowledge priors are present. This discrepancy highlights a miscalibration in VLM self‑knowledge that differs from human cognition.

By Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li, Kyle Mahowald, Michal Golovanevsky, William Rudman

llms multimodal benchmarks

arXiv Machine Learning

Jul 9

Unveiling the Visual Counting Bottleneck in Vision-Language Models

arXiv:2605. 30170v2 Announce Type: replace-cross Abstract: While Large Vision-Language Models (VLMs) excel at interpolation, they suffer catastrophic failures in systematic generalization, most notably in visual counting.

By Xingzhou Pang, Yifan Hou, Junling Wang, Mrinmaya Sachan

llms multimodal benchmarks

arXiv AI

Jul 7

Human-like Object Grouping in Self-supervised Vision Transformers

arXiv:2603. 13994v2 Announce Type: replace-cross Abstract: Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties.

By Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang, Nikolaus Kriegeskorte, Gregory Zelinsky

llms computer-vision efficiency benchmarks safety

Hugging Face Trending Papers

Sep 23

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a frontier-centric design: each model is reduced to a...

llms multimodal benchmarks

arXiv Computation and Language

Sep 24

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

PRISM‑VLM is a new benchmark for compact vision‑language models that evaluates each item across seven axes—task quality, behavioral robustness, and capability bottlenecks—rather than collapsing performance into a single accuracy score. It aggregates these axes into a single PScore while also providing per‑axis profiles, revealing differences that single‑axis benchmarks miss, such as sycophancy. The benchmark draws on items from fifteen public datasets and will be released with its full pipeline, prompts, and annotations.

By Sanghee Park, Kee-Eung Kim

llms multimodal benchmarks

arXiv AI

Jun 8

The Geometry of Representational Failures in Vision Language Models

arXiv:2602. 07025v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) exhibit puzzling failures in multi-object visual tasks, such as hallucinating non-existent elements or failing to identify the most similar objects among distractions.

By Daniele Savietto, Declan Campbell, Andr\'e Panisson, Marco Nurisso, Giovanni Petri, Jonathan D. Cohen, Alan Perotti

llms multimodal

arXiv AI

Jul 16

Beyond Color Geometry: Evaluating Human-Like Color Representations in Vision Models

arXiv:2607. 13647v1 Announce Type: cross Abstract: Do vision models see colors the way humans do?

By Ayan Igali, Pakizar Shamoi

llms computer-vision safety

arXiv AI

Aug 18

NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

arXiv:2608. 15425v1 Announce Type: cross Abstract: Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges in human infants before language acquisition, remains poorly understood in current models, as existing counting benchmarks entangle numerosity with correlated visual factors.

By Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, Angelo Cangelosi, Zhegong Shangguan

llms multimodal benchmarks

arXiv Computer Vision

Sep 16

Extremely coarse learning objectives induce human-aligned representations in AI vision models

arXiv:2605.05556v2 Announce Type: replace Abstract: Artificial neural networks trained on visual tasks develop internal representations resembling those of the primate visual system, a discovery that...

By Yash Mehta, Michael F. Bonner

llms rag safety

arXiv AI

Sep 10

Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

arXiv:2609.09124v1 Announce Type: cross Abstract: Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, inf...

By Xiaofu Chen, Stella Frank, Yova Kementchedjhieva

llms multimodal

Hugging Face Trending Papers

Aug 17

Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models

Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the...

llms multimodal benchmarks safety

arXiv AI

Sep 16

Same Answer, Different Representations: Hidden instability in VLMs

arXiv:2602.06652v2 Announce Type: replace Abstract: The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions r...

By Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena, Aryo Pradipta Gema, Wai-Chung Kwan, Fazl Barez, Maria Sofia Bucarelli, Fabrizio Silvestri, Pasquale Minervini

llms rag multimodal benchmarks safety

Related stories