← Back to all news
arXiv Machine Learning By Aras Kavuncu

LSTM-UT and Recurrent-Depth Transformers on Cellular Automata

Read the original on arXiv Machine Learning →

The paper compares three recurrent-depth transformer architectures—Block Universal Transformer (BUT), CoTFormer, and a new LSTM Universal Transformer (LSTM‑UT)—on cellular automata and delayed‑recall tasks. BUT consistently extrapolates to unseen depths better than CoTFormer, though its accuracy eventually falls. LSTM‑UT, which uses bounded gated memory, outperforms both baselines in depth extrapolation and delayed recall, demonstrating the value of gated memory for repeated computation and retrieval.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 16

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

The paper proposes a lightweight recurrent memory module inserted between the lower and upper halves of a 6‑layer decoder‑only transformer. This module, which uses cross‑attention to observe hidden states, a GRU to update a persistent state, and gated addition to modulate subsequent layers, adds only 3.7% more parameters. It reduces evaluation loss by 28.5% and narrows the generalization gap, with ablations showing the benefit comes solely from the memory topology rather than auxiliary losses.

By Eduardo Novaes Hering
llms
More like this →
arXiv Machine Learning
6d ago

LSTM-Style Looped Transformers with Shared KV Caches

The paper introduces an LSTM-style transformer that shares a single key–value cache across its recurrent core, reducing parameters by 40.4% compared to an untied model while keeping forward FLOPs similar. It aligns parallel and continuation training passes with a consistency loss, eliminating 81–89% of the continuation‑loss penalty on FineWeb. Experiments across multiple widths and token budgets show that weight tying incurs the smallest loss cost at lower token budgets, highlighting its potential for data‑ and storage‑constrained settings.

By Andreas Kirsch
llms
More like this →