arXiv Machine Learning
Aug 5
Attention is Case-Sensitive
arXiv:2608. 03711v1 Announce Type: cross Abstract: In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text.
Attention is Case-Sensitive
In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large Language Models (LLMs) exhibit an analogous property: letter casing modulates internal attention allocation.
Related stories
arXiv AI
Jul 28
Disentangling Semantic Attention from Structural Bias in the Attention Manifold
arXiv:2607. 24017v1 Announce Type: cross Abstract: The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws.
arXiv AI
Jul 7
Attending to Multimodal Generation One Token at a Time
arXiv:2607. 03738v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context.
arXiv AI
Jun 16
When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
arXiv:2602. 00344v2 Announce Type: replace-cross Abstract: While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens.
arXiv AI
Aug 6
Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.
arXiv AI
Jun 11
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.
arXiv AI
Jun 24
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
arXiv:2605. 08245v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input.
arXiv AI
Jun 8
Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
arXiv:2605. 20950v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference.
arXiv AI
Jun 15
Mirage Probes: How Vision Models Fake Visual Understanding
arXiv:2606. 13870v1 Announce Type: cross Abstract: Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided.
arXiv AI
Jul 13
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
arXiv AI
1d ago
Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
arXiv:2509. 06461v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable success across diverse visual tasks, yet their performance degrades in complex visual environments.
arXiv AI
Jun 16
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
arXiv:2603. 24058v2 Announce Type: replace-cross Abstract: Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis.