arXiv Machine Learning
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 14
RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
RunningTensor generalizes linear attention and state‑space models by extending the recurrent memory from a second‑order tensor (matrix) to an order‑o tensor. The memory is updated via a rank‑1 outer product and read by contracting with o‑1 vector queries, with order‑2 recovering linear attention. Experiments on synthetic associative recall and real language tasks show that RunningTensor improves memory capacity from O(W²) to O(Wᵒ) and outperforms existing baselines.
arXiv Machine Learning
Jun 2
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
arXiv:2606. 01294v1 Announce Type: cross Abstract: Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks.
arXiv Machine Learning
Jun 25
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
arXiv:2606. 25156v1 Announce Type: new Abstract: Modern large language models based on softmax scaled-dot-product attention are constrained by their training sequence length: as the key-value sequence grows, softmax probability mass can dilute across a wider distribution, inducing activation shift and long-context performance collapse.
arXiv AI
1d ago
Switching Linear Attention
Switching Linear Attention (SwiLA) is a new sequence layer that improves upon standard softmax attention by maintaining a fixed-size recurrent state while enhancing representational capacity. It derives its recurrence from a test-time regression framework, using online expectation-maximization in a mixture of linear regressions model. In various benchmarks—including associative recall, in-context language learning, and language modeling—SwiLA achieves strong performance, narrowing the gap to softmax attention and even surpassing it in some settings.
arXiv AI
Aug 18
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
arXiv AI
Aug 6
Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
arXiv:2608. 02347v2 Announce Type: replace Abstract: Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling.