← Back to all news
arXiv Computation and Language By Anders Gj{\o}lbye, Sebastian Ray Mason, Stella Frank, Lars Kai Hansen

Vision Transformers Learn Word Meaning from Images Alone

Read the original on arXiv Computation and Language →

The study demonstrates that large DINOv3 vision transformers, trained solely on images without any language supervision, can organize rendered words in their embedding space according to meaning. By deriving word representations from differences between word images and matched controls, the authors show that semantically related words cluster together, brand names group by industry, and country-capital pairs exhibit consistent vector offsets. These representations align with visual object categories and share structure with text and speech encoders, indicating that image-only training can impart both visual and semantic understanding of words.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 17

Using OCR Heads to Verbalize Image Semantics

The paper investigates how vision‑language models (VLMs) perform optical character recognition (OCR) by identifying attention heads that are causally necessary for OCR across four models. These heads are shown to be general‑purpose, producing interpretable semantic features for any image token, such as recognizing the word "bike" or the concept "feathers". By collapsing the heads’ attention weights into a verbalization lens transformation, the authors reveal that image representations align with language from early layers and can even be used to edit non‑word concepts in images, demonstrating the broader utility of this subspace.

By Sheridan Feucht, Benno Krojer, Sarah Wang, Henry Abrahamsen, Byron C. Wallace, David Bau
computer-visionsafety
More like this →
arXiv AI
Oct 7

Metonymic Circuits for Abstract Concept Grounding in Vision Transformers

The paper investigates how Vision Transformers (ViTs) ground abstract concepts such as "angry" when training data provide limited direct evidence. It proposes a metonymic grounding mechanism where abstract predictions are driven by concrete, interpretable anchor concepts like "fire" that bridge visual signals to abstract semantics. Using Transcoders on CLIP and DINO encoders, the authors recover intermediate features linked to concrete concepts and trace their contributions in circuits that recognize abstract concepts, revealing structured metonymic circuits and distinct perceptual-to-textual routes for images with rendered text.

By Jing Ding, Ziqiao Ma, Jiayuan Mao, Joyce Chai, Freda Shi
llms
More like this →