arXiv AI
Jul 29
Raven: High-Recall Sequence Modeling with Sparse Memory Routing
arXiv:2607. 25357v1 Announce Type: cross Abstract: Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory.
RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
RAM‑Net introduces a linear‑time sequence model that replaces dense access to a shared recurrent state with sparse, address‑based access. The model arranges the state as a fixed‑size array of independent slots and uses an Address Decoder to map each token’s key or query to a small subset of slots, thereby reducing inter‑token interference. Empirical results show that RAM‑Net outperforms strong recurrent baselines on fine‑grained long‑range retrieval, achieves the lowest perplexity in commonsense reasoning, and accesses far fewer state elements per step than competing models.
Related stories
arXiv Machine Learning
Aug 14
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.
arXiv Machine Learning
Aug 4
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
arXiv:2608. 02032v1 Announce Type: new Abstract: Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures.
Hugging Face Trending Papers
Aug 3
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states.
arXiv AI
3d ago
Write What Matters: Disentangling Token Selection from Slot Allocation in Recurrent Memory
arXiv:2610.34180v1 Announce Type: cross Abstract: Long-sequence models must retain earlier information. Full attention keeps all past tokens but grows expensive, and recurrent-memory models compress...
arXiv Machine Learning
Jul 9
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
arXiv:2607. 07386v1 Announce Type: new Abstract: Linear attention models allow a fixed state size and a fixed amount of compute per token.
arXiv Machine Learning
Jun 30
Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory
arXiv:2606. 28876v1 Announce Type: cross Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory.
arXiv Machine Learning
Sep 22
ARM: Attention with Routed-Memory for Learnable Sparse Control
arXiv:2609.24417v1 Announce Type: new Abstract: Despite advances in long-context inference, large language models (LLMs) remain fundamentally limited by the key-value (KV) caching mechanisms that are...
arXiv Computation and Language
Oct 1
The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends
arXiv:2609.39661v1 Announce Type: new Abstract: Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value c...
arXiv AI
Oct 15
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
arXiv Machine Learning
Aug 18
DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding
arXiv:2608. 15533v1 Announce Type: cross Abstract: Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates.
arXiv AI
Aug 18
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.