← Back to all news
arXiv Computation and Language By Eduardo Novaes Hering

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

Read the original on arXiv Computation and Language →

The paper proposes a lightweight recurrent memory module inserted between the lower and upper halves of a 6‑layer decoder‑only transformer. This module, which uses cross‑attention to observe hidden states, a GRU to update a persistent state, and gated addition to modulate subsequent layers, adds only 3.7% more parameters. It reduces evaluation loss by 28.5% and narrows the generalization gap, with ablations showing the benefit comes solely from the memory topology rather than auxiliary losses.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
5d ago

Dynamical Memory Transformer: Selective Recurrent Memories with Constant Cost Attention

The Dynamical Memory Transformer (DMT) is a hybrid architecture that decouples sequence encoding from information retrieval. It employs multiple memory units—each a dynamical system with a complex diagonal recurrent matrix—updated via an input‑dependent leak rate, while the decoder uses attention to read these units without accessing the full token history. This design enables an associative scan with linear total work and logarithmic parallel depth, keeping recurrent states and cache sizes constant during autoregressive inference. DMT outperforms Transformers on several cognitive tasks and matches their language‑modeling performance, though it lags behind Mamba in language modeling; it also shows superior length extrapolation on selective copy tasks.

By Yannis Bendi-Ouis (Mnemosyne), Xavier Hinaut (Mnemosyne)
llms
More like this →
arXiv Machine Learning
Aug 27

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi
llmsefficiency
More like this →