The paper adapts Stevens's power law to assess how AI models, specifically GPT‑5.5, implicitly interpret visualizations. In a pilot study, the model first estimates the magnitude of a reference visual representation and then compares subsequent images of the same representation, without access to legends or colormap names. The evaluation covers twelve visual variables, aiming to make algorithmic reading of visual encodings measurable, comparable to human perception, and more transparent.
arXiv:2610.08365v1 Announce Type: new Abstract: We adapt Stevens's power law to measure the innate ability of AI models to read visualizations, which can reveal the built-in perceptual mechanisms of...
We adapt Stevens's power law to measure the innate ability of AI models to read visualizations, which can reveal the built-in perceptual mechanisms of algorithmic models. In our pilot study, models se...
arXiv:2505.17202v2 Announce Type: replace-cross Abstract: Data visualization understanding, which requires integrating visual, numerical, and linguistic inputs, poses fundamental challenges to multim...
arXiv:2608. 10195v1 Announce Type: cross Abstract: Human vision organizes what it sees into wholes: same-colored points group into series, similar marks cohere into categories, and shapes complete into recognizable objects.
The paper introduces the Graded Color Attribution (GCA) dataset, a benchmark that tests whether Vision‑Language Models (VLMs) and humans can articulate and follow a threshold rule for labeling objects by color. In experiments, humans consistently adhere to their stated rules, while VLMs—despite accurately estimating color coverage—often violate their own introspective rules, especially when world‑knowledge priors are present. This discrepancy highlights a miscalibration in VLM self‑knowledge that differs from human cognition.
arXiv:2602. 15278v2 Announce Type: replace-cross Abstract: The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs).
arXiv:2609.09124v1 Announce Type: cross Abstract: Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, inf...
The paper introduces Embedded Stroop, a diagnostic test that embeds text prompts directly into images to study interference in multimodal large language models (MLLMs). Using the What-Color-Is-the-Text (WCIT) benchmark, which tests 59 fine‑grained colors in standard, flipped, and masked conditions, the authors evaluate 16 models and find that while exact color accuracy is low (6.3%), models still recognize coarse color families (38.4%) but frequently hallucinate the embedded word instead of the true color (Stroop Hallucination Rate of 21.6%). Masking or flipping the embedded text reduces hallucinations, indicating that semantic legibility can dominate visual color perception in MLLMs.
The paper investigates how vision encoders and Vision‑Language Models (VLMs) encode conceptual information by using canonical color as a test case. By creating a dataset of objects with canonical colors and probing encoders with both color and grayscale images, the authors show that canonical color can still be decoded from grayscale inputs and is linked to predicted object identity. They further demonstrate that post‑training of VLMs can significantly influence color decodability within the vision encoder, suggesting that canonical color is a useful tool for tracing conceptual semantics in these models.
arXiv:2608. 15736v1 Announce Type: new Abstract: Foundation models (FMs) increasingly support multimodal and geospatial reasoning, yet it remains unclear whether cartographic principles designed for human perception are equally effective for machines.
The paper introduces Humanity's Sixth Sense (HSS), a benchmark designed to evaluate intuitive visual reasoning in multimodal large language models. HSS covers a wide range of image and video inputs, organizes them under a structured taxonomy, and pairs each with human‑written prompts that probe implicit temporal, spatial, social, and abstract cues that humans can infer at a glance. Current state‑of‑the‑art models, including GPT‑6‑astra, achieve far lower accuracy than humans (53.6 % vs. 93.1 %), highlighting a significant gap in intuitive visual understanding.
The paper introduces BedBench, a benchmark of thirteen procedurally generated tasks designed to test vision‑language models’ ability to recover and consistently use exact 3D object properties. Across twelve open‑weight and four frontier models, the study finds a stark gap: humans achieve 93.3 % accuracy on per‑color counting, while the best model scores only 7.2 %. Even targeted fine‑tuning can raise a model’s accuracy to 68.6 % but leaves its answers about the most common colors inconsistent with its own per‑color counts, highlighting that recovering a property does not guarantee its consistent use.