Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention

The Dynamical Memory Transformer (DMT) is a hybrid architecture that decouples sequence encoding from information retrieval. It employs multiple memory units—each a dynamical system with a complex diagonal recurrent matrix—updated via an input‑dependent leak rate, while the decoder uses attention to read these units without accessing the full token history. This design enables an associative scan with linear total work and logarithmic parallel depth, keeping recurrent states and cache sizes constant during autoregressive inference. DMT outperforms Transformers on several cognitive tasks and matches their language‑modeling performance, though it lags behind Mamba in language modeling; it also shows superior length extrapolation on selective copy tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms