Recurrent Looped Transformer

The Recurrent Looped Transformer (RLT) divides its layers into a parallel causal encoder and a recurrent decoder, allowing the computation path to grow with sequence length while keeping a fixed per-token cost. Experiments on six algorithmic tasks show that RLT outperforms a standard eight‑layer Transformer, achieving perfect generalization on parity up to 256 bits and high accuracy on swap‑based permutation tracking and modular arithmetic beyond training lengths. Ablation studies confirm that the recurrent feedback is essential for these gains, with chunked updates preserving performance on parity but reducing accuracy on permutation tracking.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• rag