arXiv Machine Learning
Continuum Attention
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
arXiv AI
Aug 11
Linearized 2-Simplicial Attention
arXiv:2608. 09307v1 Announce Type: new Abstract: We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over one token axis takes the same form as ordinary softmax attention.
arXiv AI
Jun 2
Forget Attention: Importance-Aware Attention Is All You Need
arXiv:2606. 02332v1 Announce Type: new Abstract: Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling.
arXiv Machine Learning
2d ago
Interdomain Attention: Beyond Token-Level Key-Value Memory
The paper introduces Interdomain Attention (IA), a hybrid architecture that blends softmax attention with recurrent memory within a single layer, leveraging a kernel regression perspective to compress keys and values into a state‑space model (SSM) memory. IA reconstructs the attention matrix implicitly from memory states, preserving the computational efficiency of SSMs while achieving performance gains. Experiments on models ranging from 125 M to 1.3 B parameters show IA surpasses an S4D baseline at matched memory size, outperforms standard Transformers on perplexity and common‑sense reasoning, and matches or exceeds SSM‑based models on genomic downstream tasks.
arXiv Computation and Language
Sep 14
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...
arXiv Machine Learning
4d ago
Pretraining Transformers with Quantized Softmax in Attention
The paper investigates how quantizing the softmax operation in Transformer attention affects pretraining. It explores K‑interval attention, varying grid calibration, interpolation versus hard rounding, and surrogate placement relative to normalization, deriving backward rules for each case. Experiments on a 124M‑parameter model show that choices like fixed‑window calibration with a post‑normalization surrogate or a pre‑normalization surrogate can reduce validation loss gaps compared to standard softmax.