arXiv Machine Learning

Allocating Recurrent Compute in Looped Language Models

The paper proposes MixerLoop, a looped language model that repeats only the Gated DeltaNet mixer while applying the dense feed‑forward network (FFN) once, instead of repeating the entire layer stack. Using Iterative Transport Rank (ITR) to measure cross‑position influence, the authors show that MixerLoop retains most of the performance gains of full recurrence while reducing FLOPs by 45.9%. Experiments on 15M and 110M parameter models demonstrate that MixerLoop outperforms full‑block recurrence on aggregate CORE metrics.

arXiv AI
Sep 3

CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models

The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.

By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
arXiv AI
Sep 18

Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models

The paper critiques the common practice of evaluating depth usage in depth‑recurrent language models by truncating depth during inference and measuring performance decline. It argues that this method conflates three distinct effects—fewer block applications, reduced computation, and an out‑of‑distribution readout—yet is usually interpreted as measuring only the second. To address this, the authors introduce the Depth Control Protocol (DCP), a suite of positive and negative controls that isolate each factor, along with a training intervention to confirm causality, specifically tailored for depth‑wise weight‑sharing architectures.

By Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh Dung
arXiv Machine Learning
Sep 4

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

RecurTrace introduces adaptive latent reasoning for language models by allowing each looped layer to attend to its own past states and by using a halting head to decide when to stop iterating. This approach overcomes two limitations of prior latent recurrence methods: limited access to earlier computations and a fixed loop count that mismatches input difficulty. In experiments on MathQA, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, outperforming fixed‑depth baselines and other adaptive methods, and it also improves generation accuracy across a range of model sizes.

By Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu, Junyu Wang, Zhizheng Wu
arXiv Machine Learning
Sep 14

Temporal Recurrence Favors Fewer Layers

The paper investigates how temporal recurrence affects the required depth of neural networks in streaming tasks. By treating depth, expert width, and parallel experts as a compute‑allocation problem, the authors compare recurrent and non‑recurrent models across various compute budgets. Experiments on Sokoban and FineWeb language modeling show that recurrence shifts the optimal compute allocation toward fewer layers while maintaining or improving performance.

By Ivan Anokhin, Johan Obando-Ceron, Irina Rish, Sebastian Risi
arXiv Computation and Language
Oct 21

Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

The Asymmetric Looped Transformer (AsymLoop) separates context encoding from state refinement by using a single-pass causal encoder to produce fixed global key-value caches and a weight-tied recurrent decoder to generate local key-values from its evolving state. Joint attention over both sources allows each update to combine persistent contextual information with the current intermediate state. Experiments on FineWeb-Edu across models from 60M to 1B show lower validation perplexity and higher downstream accuracy than full-loop models, with a 1B model improving zero-shot commonsense reasoning accuracy from 49.58% to 50.85%.

By Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang, Zhouhan Lin
arXiv Computation and Language
Oct 12

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

The paper introduces Elastic Latent Recurrence (ELR), a training recipe that enables looped language models to match the accuracy of non-recurrent models at a single loop while allowing additional loops to improve accuracy with more compute. ELR achieves this through gated state updates with learned step sizes, mixed loop counts during training, and conditioning on loop index and budget. The method also reduces training and inference overhead in Mixture-of-Experts models and produces a single checkpoint, Koi‑4B, that performs competitively with state‑of‑the‑art models on multiple benchmarks.

By Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada
arXiv Machine Learning
Aug 4

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.

By Benjamin L. Badger
arXiv Machine Learning
Oct 19

Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

The paper investigates looped language models that reuse Transformer blocks across logical depth, separating recurrence placement, input injection, and recurrent state design. Experiments at matched parameter scale, logical depth, and token budgets show that a prelude‑recurrent‑block‑coda architecture improves context extraction and reasoning, while input injection boosts knowledge retrieval. The Huginn‑style design, especially at 730 M parameters and 336 B tokens, outperforms the Ouro architecture on all ten benchmarks, and its advantages transfer to Mixture‑of‑Experts models, achieving superior performance with fewer resident parameters.

By Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma
arXiv Machine Learning
Oct 9

TraceRelay: Attention-Aligned Recurrence over Rolling Traces

TraceRelay is an attention‑aligned recurrent architecture that distributes persistent representations over a rolling sequence of low‑dimensional traces. It uses right‑looking attention to form increments from lower‑layer representations, delays delivery until all attended inputs are in the causal past, and accumulates these increments with a fixed additive phase recurrence. Experiments on 36 small‑model runs show that recurrent phase inheritance yields high accuracy on Equal Repeats (up to 99.69%) but accuracy drops beyond the training range, and that the optimal size of recurrent traces varies across tasks such as Dyck closing‑type prediction and causal Most‑Freq generation.

By Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung