Hugging Face Blog
Perceiver IO: a scalable, fully-attentional model that works on any modality
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv Computer Vision
Aug 27
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
RAVE (Re-Allocating Visual Attention) is a lightweight pair‑gating mechanism that adds a learned query‑key bias to pre‑softmax attention scores over visual keys, derived from pre‑RoPE query and key features. It requires no architectural changes to the backbone and can be trained end‑to‑end with the rest of the model. Across multiple multimodal benchmarks, RAVE improves standard attention by an average of 3 points, especially on perception‑intensive tasks such as multilingual OCR, chart understanding, document VQA, and scene text VQA.
arXiv AI
Jun 6
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
arXiv:2606. 05843v1 Announce Type: cross Abstract: While Multimodal Large Language Models (MLLMs) demonstrate remarkable proficiency on complex vision-language tasks, the mechanisms by which they extract query-relevant visual features from complex, noisy contexts remain opaque.
arXiv Computation and Language
Aug 31
Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs
The paper introduces Semantic Head Specialization (SHS), a phenomenon where Vision Transformer (ViT) attention heads specialize as either object- or background-focused, most evident under full attention. It proposes the SHS-Index to quantify this specialization, demonstrating its ability to distinguish full-attention from chunk-window ViTs and its strong correlation with downstream benchmark performance. Leveraging insights into window interaction, token serialization, and local softmax allocation, the authors design Ariadne Attention, a hybrid attention mechanism that matches full-attention performance on 22 image and video tasks while reducing attention compute by 6.5×.
arXiv Machine Learning
Jul 16
Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models
arXiv:2607. 13395v1 Announce Type: new Abstract: The pursuit of autonomously self-improving models has attracted growing interest in the era of large-scale foundation models.
arXiv AI
Jun 30
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.
arXiv Computation and Language
Aug 25
Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
arXiv:2606.14782v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning but incur large KV caches and high decoding latency with lo...