arXiv Computation and Language
You Need Better Attention Priors
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
4d ago
How Local Mixing Encodes Relative Position in Global NoPE Attention
arXiv:2609.38109v1 Announce Type: cross Abstract: The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of...
arXiv Machine Learning
Jul 2
Information-Regularized Attention for Visual-Centric Reasoning
arXiv:2607. 00434v1 Announce Type: cross Abstract: Vision-language models (VLMs) have become a paradigm for multimodal learning, yet remain unstable due to object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning.
arXiv Machine Learning
Sep 24
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
arXiv Computer Vision
Aug 27
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
RAVE (Re-Allocating Visual Attention) is a lightweight pair‑gating mechanism that adds a learned query‑key bias to pre‑softmax attention scores over visual keys, derived from pre‑RoPE query and key features. It requires no architectural changes to the backbone and can be trained end‑to‑end with the rest of the model. Across multiple multimodal benchmarks, RAVE improves standard attention by an average of 3 points, especially on perception‑intensive tasks such as multilingual OCR, chart understanding, document VQA, and scene text VQA.
arXiv Computer Vision
2d ago
Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
arXiv:2605.12491v2 Announce Type: replace Abstract: Vision Transformers (ViTs) learn rich visual-semantic representations through all-to-all self-attention among patch tokens. However, this design im...
arXiv Computer Vision
2d ago
Subtoken Vision Transformer for Fine-grained Recognition
arXiv:2607.09086v2 Announce Type: replace Abstract: We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transfo...