arXiv AI
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 14
Extending LLM Context via Associative Recurrent Memory
arXiv:2607. 11614v1 Announce Type: cross Abstract: Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling.
arXiv AI
Aug 18
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
arXiv Machine Learning
Jun 11
On Subquadratic Architectures: From Applications to Principles
arXiv:2606. 12364v1 Announce Type: new Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost.
arXiv AI
Jul 9
Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization
arXiv:2604. 01577v3 Announce Type: replace-cross Abstract: We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.
arXiv AI
Jun 4
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
arXiv AI
Oct 7
Continuous Memory Machines
The paper introduces the Continuous Memory Machine (CMM), a recurrent neural network that uses separate matrix-valued short‑term and long‑term memory states. Short‑term memory tracks recent neural activity with neuron‑level models for rapid computation, while long‑term memory stores information for later use; both are jointly updated by a Transformer that allows bidirectional read‑write operations. Experiments on algorithmic, in‑context learning, and recurrent reasoning tasks show that CMM outperforms many baselines and generalizes better than previous memory‑augmented networks, while maintaining interpretable attention patterns from the Continuous Thought Machine.