arXiv:2608. 15736v1 Announce Type: new Abstract: Foundation models (FMs) increasingly support multimodal and geospatial reasoning, yet it remains unclear whether cartographic principles designed for human perception are equally effective for machines.
arXiv:2607. 17999v1 Announce Type: new Abstract: Spatial understanding is crucial for foundation models (FMs), and maps have long helped humans organize and reason about geographic information.
Spatial understanding is crucial for foundation models (FMs), and maps have long helped humans organize and reason about geographic information. This study examines whether choropleth maps remain useful for machine spatial understanding when models can directly process structured geodata.
arXiv:2510. 04514v3 Announce Type: replace Abstract: Recent multimodal LLMs have shown promise in chart-based visual question answering, but their performance declines sharply on unannotated charts-those requiring precise visual interpretation rather than relying on textual shortcuts.
The paper introduces Humanity's Sixth Sense (HSS), a benchmark designed to evaluate intuitive visual reasoning in multimodal large language models. HSS covers a wide range of image and video inputs, organizes them under a structured taxonomy, and pairs each with human‑written prompts that probe implicit temporal, spatial, social, and abstract cues that humans can infer at a glance. Current state‑of‑the‑art models, including GPT‑6‑astra, achieve far lower accuracy than humans (53.6 % vs. 93.1 %), highlighting a significant gap in intuitive visual understanding.
arXiv:2602. 18600v5 Announce Type: replace Abstract: Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI).
The paper adapts Stevens's power law to assess how AI models, specifically GPT‑5.5, implicitly interpret visualizations. In a pilot study, the model first estimates the magnitude of a reference visual representation and then compares subsequent images of the same representation, without access to legends or colormap names. The evaluation covers twelve visual variables, aiming to make algorithmic reading of visual encodings measurable, comparable to human perception, and more transparent.
The paper introduces Embedded Stroop, a diagnostic test that embeds text prompts directly into images to study interference in multimodal large language models (MLLMs). Using the What-Color-Is-the-Text (WCIT) benchmark, which tests 59 fine‑grained colors in standard, flipped, and masked conditions, the authors evaluate 16 models and find that while exact color accuracy is low (6.3%), models still recognize coarse color families (38.4%) but frequently hallucinate the embedded word instead of the true color (Stroop Hallucination Rate of 21.6%). Masking or flipping the embedded text reduces hallucinations, indicating that semantic legibility can dominate visual color perception in MLLMs.
The paper introduces ImagiRep, a benchmark that tests spatial reasoning using seven question types across two settings, offering RGB and six alternative representations of imagined states. It shows that accurate RGB generation does not guarantee correct spatial reasoning and that different questions benefit from different representations such as depth, panoramic views, or top‑down geometry. Building on these insights, the authors present GEARS, a lightweight strategy that adaptively selects representations at test time, outperforming existing methods and demonstrating that reasoning about unseen states requires more than just RGB generation.
arXiv:2511. 17731v2 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting has proven remarkably effective for eliciting complex reasoning in large language models (LLMs).
arXiv:2605. 09883v2 Announce Type: replace-cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding?
The paper introduces BedBench, a benchmark of thirteen procedurally generated tasks designed to test vision‑language models’ ability to recover and consistently use exact 3D object properties. Across twelve open‑weight and four frontier models, the study finds a stark gap: humans achieve 93.3 % accuracy on per‑color counting, while the best model scores only 7.2 %. Even targeted fine‑tuning can raise a model’s accuracy to 68.6 % but leaves its answers about the most common colors inconsistent with its own per‑color counts, highlighting that recovering a property does not guarantee its consistent use.
arXiv:2610.32473v1 Announce Type: cross Abstract: Dominant color extraction is a fundamental task in computer vision, with applications in design, image retrieval, and content recommendation. However...