arXiv AI

Metonymic Circuits for Abstract Concept Grounding in Vision Transformers

The paper investigates how Vision Transformers (ViTs) ground abstract concepts such as "angry" when training data provide limited direct evidence. It proposes a metonymic grounding mechanism where abstract predictions are driven by concrete, interpretable anchor concepts like "fire" that bridge visual signals to abstract semantics. Using Transcoders on CLIP and DINO encoders, the authors recover intermediate features linked to concrete concepts and trace their contributions in circuits that recognize abstract concepts, revealing structured metonymic circuits and distinct perceptual-to-textual routes for images with rendered text.

arXiv Computer Vision

Oct 9

From What to Which: Decoding Modifier Grounding in Frozen MLLMs

The paper investigates whether frozen Multimodal Large Language Models (MLLMs) encode grounding information about specific visual instances within the tokens they generate, including modifiers such as attributes, spatial expressions, and relational terms. The authors introduce OTTER, a lightweight supervised probe that applies Optimal Transport to align generated tokens with visual regions, producing compact grounding maps. Experiments demonstrate that modifier tokens, especially spatial terms, carry instance‑discriminative visual information, that object nouns also encode referential cues, and that the recovered grounding remains robust to context changes and generalizes to free generation and cross‑dataset transfer.

By Barbara Toniella Corradini (AI for Good), Caterina Gallegati (University of Siena, Italy), Ludovica Genovese (AI for Good), Vittorio Murino (AI for Good)

llms multimodal

arXiv AI

Jun 30

Can Machines Really See Objects in Images? A Study Based on Syntactic Distance and Visual Self-Referential Instances

arXiv:2606. 29416v1 Announce Type: cross Abstract: Can a vision model truly see an object, or does it only fit surface-level visual cues?

By Xingyu Peng, Junran Wu, Yue Hou, Zhongliang Qiao, Jiaheng Liu, Shangzhe Li, Jichang Zhao, Wenjun Wu, Xianglong Liu, Yongxin Tong, Li Dong, Ke Xu

llms

arXiv Computer Vision

Sep 24

Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning

arXiv:2609.28187v1 Announce Type: new Abstract: Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, y...

By Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski

llms safety

arXiv AI

Sep 2

Causal Probing for Internal Visual Representations in Multimodal Large Language Models

arXiv:2605.05593v2 Announce Type: replace Abstract: Despite the remarkable success of Multimodal Large Language Models (MLLMs) across diverse tasks, the internal mechanisms governing how they encode...

By Zehao Deng, Tianjie Ju, Zheng Wu, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang

llms multimodal

arXiv AI

Jun 8

The Geometry of Representational Failures in Vision Language Models

arXiv:2602. 07025v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) exhibit puzzling failures in multi-object visual tasks, such as hallucinating non-existent elements or failing to identify the most similar objects among distractions.

By Daniele Savietto, Declan Campbell, Andr\'e Panisson, Marco Nurisso, Giovanni Petri, Jonathan D. Cohen, Alan Perotti

llms multimodal

arXiv Computer Vision

Oct 5

Decoding the Functional Roles of Register and High-Norm Patch Tokens in Vision Transformers

arXiv:2610.03698v1 Announce Type: new Abstract: Self-supervised Vision Transformers (ViTs), such as DINOv2, learn rich visual representations, but the functions of their internal tokens remain poorly...

By Neel Varma, Andrew Rufail, Dipika Khullar, Vasu Sharma

llms safety

arXiv Computation and Language

Oct 12

Vision Transformers Learn Word Meaning from Images Alone

The study demonstrates that large DINOv3 vision transformers, trained solely on images without any language supervision, can organize rendered words in their embedding space according to meaning. By deriving word representations from differences between word images and matched controls, the authors show that semantically related words cluster together, brand names group by industry, and country-capital pairs exhibit consistent vector offsets. These representations align with visual object categories and share structure with text and speech encoders, indicating that image-only training can impart both visual and semantic understanding of words.

By Anders Gj{\o}lbye, Sebastian Ray Mason, Stella Frank, Lars Kai Hansen

llms rag safety

arXiv Machine Learning

Jun 3

Visual Instruction Tuning Aligns Modalities through Abstraction

arXiv:2606. 03871v1 Announce Type: cross Abstract: Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text.

By Luis Palacios, Lorenzo Basile, Diego Doimo, Alberto Cazzaniga

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Oct 7

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

arXiv:2610.08401v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have shown outstanding performance across diverse applications, they tend to under-use visual information...

By Seulgi Kim, Zhixiong Zhang, Xinwei Zhang, Jie Ling, Ronn Shaw

llms multimodal benchmarks

arXiv AI

Sep 15

Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models

arXiv:2605.22902v2 Announce Type: replace-cross Abstract: Generative Vision-Language Models (VLMs) perform well on multimodal reasoning, but how visual inputs are transformed to text remains poorly u...

By Dimitrios Damianos, Leon Voukoutis, Georgios Skyrianos, Vassilis Katsouros, Georgios Paraskevopoulos

llms multimodal benchmarks safety

arXiv AI

Sep 17

Using OCR Heads to Verbalize Image Semantics

The paper investigates how vision‑language models (VLMs) perform optical character recognition (OCR) by identifying attention heads that are causally necessary for OCR across four models. These heads are shown to be general‑purpose, producing interpretable semantic features for any image token, such as recognizing the word "bike" or the concept "feathers". By collapsing the heads’ attention weights into a verbalization lens transformation, the authors reveal that image representations align with language from early layers and can even be used to edit non‑word concepts in images, demonstrating the broader utility of this subspace.

By Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau

computer-vision safety

arXiv Machine Learning

Jul 7

Pathways of Visual Information Flow in Vision-Language Models

arXiv:2607. 03358v1 Announce Type: cross Abstract: We study how visual information is routed in vision-language models (VLMs).

By Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

llms multimodal benchmarks

Related stories