arXiv:2510. 09379v2 Announce Type: replace Abstract: While softmax attention drives state-of-the-art performance in sequence modeling, its quadratic complexity motivates linear alternatives such as state space models (SSMs).
arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.
arXiv:2609.39661v1 Announce Type: new Abstract: Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value c...
The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner.
arXiv:2606. 12364v1 Announce Type: new Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost.
arXiv:2602. 09075v3 Announce Type: replace-cross Abstract: In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks.
arXiv:2503. 18970v4 Announce Type: replace Abstract: Structured State Space Models (SSMs) have become a prominent class of sequence models, developed against two long-standing difficulties: the sequential computation and gradient propagation limits of Recurrent Neural Networks (RNNs), and the quadratic time and memory cost of self-attention in Transformers.
arXiv:2609.06006v1 Announce Type: cross Abstract: Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state...
arXiv:2607. 09889v1 Announce Type: cross Abstract: Fixed-state sequence models compress an unbounded past into a bounded state, which caps their associative recall at roughly the state dimension; attention escapes the cap by keeping a key-value entry for every token, at quadratic compute and a cache that grows with the sequence.
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
arXiv:2609.36529v1 Announce Type: new Abstract: Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memo...
RunningTensor generalizes linear attention and state‑space models by extending the recurrent memory from a second‑order tensor (matrix) to an order‑o tensor. The memory is updated via a rank‑1 outer product and read by contracting with o‑1 vector queries, with order‑2 recovering linear attention. Experiments on synthetic associative recall and real language tasks show that RunningTensor improves memory capacity from O(W²) to O(Wᵒ) and outperforms existing baselines.
arXiv:2608. 15459v1 Announce Type: cross Abstract: Attention mechanisms have driven machine learning for a decade, from neural machine translation to language models that do general-purpose reasoning.