The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2606. 05115v1 Announce Type: cross Abstract: Children learn the meanings of words from a continuous, temporally structured stream of egocentric experience.
arXiv:2607. 16214v1 Announce Type: cross Abstract: Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivity remain unclear.
The study demonstrates that large DINOv3 vision transformers, trained solely on images without any language supervision, can organize rendered words in their embedding space according to meaning. By deriving word representations from differences between word images and matched controls, the authors show that semantically related words cluster together, brand names group by industry, and country-capital pairs exhibit consistent vector offsets. These representations align with visual object categories and share structure with text and speech encoders, indicating that image-only training can impart both visual and semantic understanding of words.
The paper introduces GAVAGAI, a method that adds a referential alignment variable to vision‑language models by solving a small assignment problem via entropic optimal transport. By allowing words to be assigned to a special NOTHING column and enforcing a capacity constraint on regions, the approach improves word‑to‑picture accuracy and localization on the SAYCam infant video corpus. Experiments show that the alignment loss boosts performance over a standard next‑token captioning baseline, especially under high non‑reference rates.
arXiv:1805. 11546v3 Announce Type: replace-cross Abstract: We examine the benefits of visual context in training neural language models to perform next-word prediction.
The paper introduces an experiment where a small masked language model (DeBERTa) is initialized with visual embeddings for tokens that correspond to image regions, following St. Augustine’s ostensive definition of word learning. The visual initialization leaves a measurable imprint that persists through training, yet it does not improve performance on most BabyLM benchmarks that test abstract grammatical knowledge. However, the seeded models show a consistent advantage in zero‑shot object‑property tasks and in a custom Visual‑Property Swap benchmark that probes color, material, size, and shape knowledge, with the advantage confined to the seeded words and transferable to newly seeded words.