arXiv AI
Jul 21
Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
arXiv:2607. 17419v1 Announce Type: cross Abstract: Linear attention promises constant-time recurrent inference but degrades sharply on associative recall.
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
arXiv:2607. 11897v1 Announce Type: new Abstract: Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory.
Related stories
Hugging Face Trending Papers
Jul 23
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well.
arXiv AI
Jul 24
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
arXiv:2607. 21000v1 Announce Type: new Abstract: Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones.
arXiv AI
Jun 10
Blurry Window Attention
arXiv:2606. 09862v1 Announce Type: cross Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.
arXiv AI
Jun 26
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
arXiv:2606. 27229v1 Announce Type: cross Abstract: Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify.
arXiv AI
Jul 14
Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
arXiv:2607. 09889v1 Announce Type: cross Abstract: Fixed-state sequence models compress an unbounded past into a bounded state, which caps their associative recall at roughly the state dimension; attention escapes the cap by keeping a key-value entry for every token, at quadratic compute and a cache that grows with the sequence.
arXiv Machine Learning
Jun 25
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
arXiv:2606. 25156v1 Announce Type: new Abstract: Modern large language models based on softmax scaled-dot-product attention are constrained by their training sequence length: as the key-value sequence grows, softmax probability mass can dilute across a wider distribution, inducing activation shift and long-context performance collapse.
arXiv Machine Learning
Jun 2
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
arXiv AI
Jul 3
A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
arXiv:2607. 02303v1 Announce Type: new Abstract: Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades.
arXiv Machine Learning
Jul 21
Overlapping Schwarz Attention: Hierarchical Attention via Domain Decomposition
arXiv:2606. 18525v2 Announce Type: replace Abstract: We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition.
arXiv Machine Learning
1d ago
DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding
arXiv:2608. 15533v1 Announce Type: cross Abstract: Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates.
arXiv AI
Jul 28
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
arXiv:2605. 09877v4 Announce Type: replace-cross Abstract: Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall.