arXiv AI
RCStat: A Statistical Framework for using Relative Contextualization in Transformers
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 3
Language Models Can Control Their Own Attention
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv AI
Jun 9
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).
arXiv AI
Jul 24
Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.
arXiv AI
Jun 26
Information-Aware KV Cache Compression for Long Reasoning
arXiv:2606. 26875v1 Announce Type: cross Abstract: Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages.
arXiv Computation and Language
Oct 5
Emergent Structure in the Marginal Attention Space of Language Models
The paper investigates how post‑softmax attention weights behave across different language models by marginalizing over query positions to create a joint token‑head space. Across more than 60 diverse LLMs, token‑wise marginal attention reveals a text‑intrinsic signal that is conserved across models, while head‑wise marginal attention yields a model‑private signature that is consistent across documents. This insight enables a practical method for estimating per‑head budgets for key‑value cache eviction, achieving competitive performance on standard benchmarks without per‑document recomputation or training.
arXiv Computation and Language
Sep 14
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...