arXiv AI

Memory in Deep Time-Series Models

Hugging Face Trending Papers
Aug 5

Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems

LLM agents that persist across sessions accumulate stored memories whose validity varies enormously by content type, yet existing memory architectures treat all memories as equally persistent and systematically contaminate retrieved context with outdated facts. We show that per-memory, type-conditioned temporal decay, a property of western scrub jay episodic memory, can be operationalized as an auto-classified coefficient $π_i$ in an external LLM-agent memory store, yielding ScrubJay-MEM: each memory is encoded as a jointly-bound What--Where--When tuple with an estimated perishability $π_i$ and utility horizon $τ_i$, retrieved by query-adaptive scoring, and revised retroactively at $O(1)$ LLM calls per update.

arXiv AI
Oct 7

Continuous Memory Machines

The paper introduces the Continuous Memory Machine (CMM), a recurrent neural network that uses separate matrix-valued short‑term and long‑term memory states. Short‑term memory tracks recent neural activity with neuron‑level models for rapid computation, while long‑term memory stores information for later use; both are jointly updated by a Transformer that allows bidirectional read‑write operations. Experiments on algorithmic, in‑context learning, and recurrent reasoning tasks show that CMM outperforms many baselines and generalizes better than previous memory‑augmented networks, while maintaining interpretable attention patterns from the Continuous Thought Machine.

By Ciaran Regan, Kai Arulkumaran, Luke Darlow, Stefania Druga, Sebastian Risi, Llion Jones
arXiv AI
Aug 7

The Impossibility Triangle of Long-Context Modeling

arXiv:2605. 05066v2 Announce Type: replace-cross Abstract: We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall).

By Yan Zhou
arXiv Machine Learning
Jul 10

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.

By Ashwin Gerard Colaco, Nada Lahjouji
arXiv AI
Oct 15

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...

By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv Machine Learning
Oct 12

HART: Hybrid Attention-RWKV Time series Foundation Model

The paper introduces HART, a Hybrid Attention-RWKV Time-series Foundation Model that alternates attention and RWKV blocks to improve long-horizon forecasting. It identifies a problem in RWKV-7 called Placeholder Idling Decay, where repeated state transitions degrade historical information during non-autoregressive multi-step forecasting. Experiments on GIFT-Eval and TIME-Benchmark show that HART outperforms pure RWKV on medium and long horizons while matching short-horizon performance, and achieves comparable long-horizon MASE to a much larger Transformer with only 36 M parameters.

By Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai
arXiv AI
Nov 2

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention

The Dynamical Memory Transformer (DMT) is a hybrid architecture that decouples sequence encoding from information retrieval. It employs multiple memory units—each a dynamical system with a complex diagonal recurrent matrix—updated via an input‑dependent leak rate, while the decoder uses attention to read these units without accessing the full token history. This design enables an associative scan with linear total work and logarithmic parallel depth, keeping recurrent states and cache sizes constant during autoregressive inference. DMT outperforms Transformers on several cognitive tasks and matches their language‑modeling performance, though it lags behind Mamba in language modeling; it also shows superior length extrapolation on selective copy tasks.

By Yannis Bendi-Ouis (Mnemosyne), Xavier Hinaut (Mnemosyne)