TraceRelay is an attention‑aligned recurrent architecture that distributes persistent representations over a rolling sequence of low‑dimensional traces. It uses right‑looking attention to form increments from lower‑layer representations, delays delivery until all attended inputs are in the causal past, and accumulates these increments with a fixed additive phase recurrence. Experiments on 36 small‑model runs show that recurrent phase inheritance yields high accuracy on Equal Repeats (up to 99.69%) but accuracy drops beyond the training range, and that the optimal size of recurrent traces varies across tasks such as Dyck closing‑type prediction and causal Most‑Freq generation.
By Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
By Ke Wan, Chen Chen
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv:2609.36337v1 Announce Type: new
Abstract: Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large d...
By David Schnurr, Felix Sarnthein, Thomas Hofmann, Imanol Schlag
The paper introduces recirculation, an inference‑time architectural enhancement for foundation models that reduces perplexity and improves accuracy on generation and reasoning tasks without adding significant latency. Recirculation adds a specific form of recurrence, enabling the model to function as a dynamical system that tracks belief states, and is distinct from chain‑of‑thought or depth‑recurrence methods. An adaptive variant requires minimal hyperparameter tuning and achieves notable gains on the Gemma3 family, including a 23% perplexity drop and a 21% accuracy increase on GSM8k.
By Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu
arXiv:2610.16301v1 Announce Type: new
Abstract: Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position...
By Helong Hu (Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China), Hongdan Pan (Maoming Administrative Service Center, Maoming, Guangdong, China), Shuiqing Hu (China Pacific Insurance)
arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.
By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan
The paper investigates how temporal recurrence affects the required depth of neural networks in streaming tasks. By treating depth, expert width, and parallel experts as a compute‑allocation problem, the authors compare recurrent and non‑recurrent models across various compute budgets. Experiments on Sokoban and FineWeb language modeling show that recurrence shifts the optimal compute allocation toward fewer layers while maintaining or improving performance.
By Ivan Anokhin, Johan Obando-Ceron, Irina Rish, Sebastian Risi
arXiv:2604. 03444v4 Announce Type: replace Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention.
By William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, DJ Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, Ashish Sabharwal
arXiv:2608. 02870v1 Announce Type: new Abstract: We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training.
By Bo Liu, Qiang Liu
arXiv:2606. 27538v1 Announce Type: cross Abstract: We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block that pre-contextualizes each token before it enters the block.
By Mahesh Godavarti
The paper proposes MixerLoop, a looped language model that repeats only the Gated DeltaNet mixer while applying the dense feed‑forward network (FFN) once, instead of repeating the entire layer stack. Using Iterative Transport Rank (ITR) to measure cross‑position influence, the authors show that MixerLoop retains most of the performance gains of full recurrence while reducing FLOPs by 45.9%. Experiments on 15M and 110M parameter models demonstrate that MixerLoop outperforms full‑block recurrence on aggregate CORE metrics.
By Ruhai Lin, Yiyang Guo, Rui-Jie Zhu, Hao Ye, Jason K. Eshraghian