← Back to all news
arXiv Computer Vision By Xi Leng, Xinhong Ma, Ziqiang Dong, Feng Zhang, Xiaoying Tang, Yang Yang, Guanjun Jiang

RAVE: Re-Allocating Visual Attention in Large Multimodal Models

Read the original on arXiv Computer Vision →

RAVE (Re-Allocating Visual Attention) is a lightweight pair‑gating mechanism that adds a learned query‑key bias to pre‑softmax attention scores over visual keys, derived from pre‑RoPE query and key features. It requires no architectural changes to the backbone and can be trained end‑to‑end with the rest of the model. Across multiple multimodal benchmarks, RAVE improves standard attention by an average of 3 points, especially on perception‑intensive tasks such as multilingual OCR, chart understanding, document VQA, and scene text VQA.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computation and Language
4d ago

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.

By Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren
llmsmultimodalbenchmarks
More like this →