arXiv Machine Learning
Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computer Vision
Aug 28
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
Hugging Face Trending Papers
Aug 27
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
The paper demonstrates that vision‑language models (VLMs) possess a small set of attention heads, called Visual Retrieval Heads (VRHs), that are causally responsible for linking text prompts to specific image regions. By adapting head‑scoring techniques from language models, the authors identify VRHs as the heads whose attention from output prediction tokens, summed over the ground‑truth referent region, most reliably indicates causal grounding. Experiments across eleven VLMs and five referring‑expression benchmarks show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect, and that VRHs generalize across diverse visual tasks and transfer across models sharing an LLM backbone.
arXiv AI
4d ago
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
The paper investigates how the modality gap— the separation between image and text representations in contrastive vision‑language models—affects different downstream tasks. By showing that a single dominant direction accounts for most of the image‑text mean separation, the authors explain why reducing or removing this gap can improve zero‑shot classification, degrade retrieval, or restore performance depending on the task. The study provides a geometric framework that clarifies when and why gap interventions should be applied in vision‑language systems.
arXiv Computation and Language
Sep 15
Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Models
arXiv:2609.15640v1 Announce Type: cross Abstract: Long-text image--text congruence scoring is increasingly important for vision-language systems that must evaluate whether detailed textual descriptio...
arXiv AI
3d ago
VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
arXiv:2605.26380v2 Announce Type: replace-cross Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...
arXiv Machine Learning
Jul 21
The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
arXiv:2607. 18237v1 Announce Type: cross Abstract: Human visual similarity judgments are context-dependent.