The paper introduces recirculation, an inference‑time architectural enhancement for foundation models that reduces perplexity and improves accuracy on generation and reasoning tasks without adding significant latency. Recirculation adds a specific form of recurrence, enabling the model to function as a dynamical system that tracks belief states, and is distinct from chain‑of‑thought or depth‑recurrence methods. An adaptive variant requires minimal hyperparameter tuning and achieves notable gains on the Gemma3 family, including a 23% perplexity drop and a 21% accuracy increase on GSM8k.
By Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu
arXiv:2604. 01577v3 Announce Type: replace-cross Abstract: We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.
By Shota Takashiro, Masanori Koyama, Takeru Miyato, Yusuke Iwasawa, Yutaka Matsuo, Kohei Hayashi
arXiv:2609.16752v1 Announce Type: new
Abstract: Cognitive Field Theory (CFT) proposes that cognition arises from memory-dressed collective dynamics that generate a persistent macroscopic cognitive fi...
By Byung Gyu Chae
arXiv:2608. 03425v1 Announce Type: new Abstract: Transformer-based architectures have dominated sequence modeling, largely due to the expressive power of attention mechanisms.
By Xiaohe Li, Yang Lu
arXiv:2502.09245v3 Announce Type: replace
Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...
By Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky, Viacheslav Sinii, Daniil Gavrilov
The paper investigates whether extra computation in recurrent Transformers should be allocated to more temporal steps or greater physical depth. Using Latent Recurrent Transformers (LRTs), the authors insert a latent thought token between vocabulary tokens, allowing each token to pass through the same $L$ layers twice while sharing parameters. Experiments on 16‑ and 20‑layer mixture‑of‑experts NanoChat backbones show that a single thought token brings a shallower model within 0.006–0.004 bits per byte of a double‑depth counterpart, recovering 67–81% of the improvement with roughly 48% fewer parameters.
By Zeyi Huang, Xuehai He, Yong Jae Lee, Yelong Shen
The paper investigates whether a global workspace—a set of verbalisable, causally potent representations—emerges in transformer models that use recurrence instead of a stack of distinct layers. Using a Jacobian lens extended with a virtual‑unrolling adapter, the authors analyze two recurrent transformer architectures, Ouro‑2.6B and Huginn‑0125, and compare them to a standard Qwen3.6‑27B baseline. They find that a workspace does form in the iterated parts of both models, but recurrence alters how it can be accessed: Ouro reconstructs workspace content in every loop and requires writes and ablations across all loops, whereas Huginn forwards content across all recurrences but limits reads, writes, and ablations to a sliding window of about two recurrences.
By Wenlong Wang, Fergal Reid
arXiv:2609.06006v1 Announce Type: cross
Abstract: Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state...
By Minh Hoang Nguyen, Huu Hiep Nguyen, Manh Nguyen, Van Dai Do, Dung Nguyen, Hung Le
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight, procedurally generated grid‑world testbed for evaluating predictive models, particularly Transformers, on tasks such as per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction. It compares byte‑level Transformer baselines with two memory‑augmented architectures, showing that a pseudo‑video spatial‑memory Transformer achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, while a generic auxiliary latent‑memory Transformer does not consistently help. The study highlights that structured, task‑aligned working memory can be more effective than merely increasing latent capacity, and positions the benchmark as a compact setting for testing architectures that couple textual interfaces to learned structured state.
By Alexey Potapov
arXiv:2608.15062v3 Announce Type: replace-cross
Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layer...
By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight testbed for evaluating how Transformers and other models learn world dynamics. The benchmark, built from procedurally generated grid worlds, includes per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction, with source‑maze‑disjoint training and validation splits to test transferable action‑conditioned dynamics. Experiments show that a pseudo‑video spatial‑memory Transformer, which initializes and updates a two‑dimensional latent workspace from the input map and action history, achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, outperforming byte‑level and unstructured‑memory baselines and suggesting that structured, task‑aligned working memory is more effective than additional latent capacity alone.
The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.
By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi