arXiv:2609.36636v1 Announce Type: new
Abstract: Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding...
By Xinlin Zhuang, Siyuan Wang, Imran Razzak, Weiyang Liu
The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.
By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
The paper critiques the common practice of evaluating depth usage in depth‑recurrent language models by truncating depth during inference and measuring performance decline. It argues that this method conflates three distinct effects—fewer block applications, reduced computation, and an out‑of‑distribution readout—yet is usually interpreted as measuring only the second. To address this, the authors introduce the Depth Control Protocol (DCP), a suite of positive and negative controls that isolate each factor, along with a training intervention to confirm causality, specifically tailored for depth‑wise weight‑sharing architectures.
By Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh Dung
RecurTrace introduces adaptive latent reasoning for language models by allowing each looped layer to attend to its own past states and by using a halting head to decide when to stop iterating. This approach overcomes two limitations of prior latent recurrence methods: limited access to earlier computations and a fixed loop count that mismatches input difficulty. In experiments on MathQA, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, outperforming fixed‑depth baselines and other adaptive methods, and it also improves generation accuracy across a range of model sizes.
By Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu, Junyu Wang, Zhizheng Wu
The paper investigates how temporal recurrence affects the required depth of neural networks in streaming tasks. By treating depth, expert width, and parallel experts as a compute‑allocation problem, the authors compare recurrent and non‑recurrent models across various compute budgets. Experiments on Sokoban and FineWeb language modeling show that recurrence shifts the optimal compute allocation toward fewer layers while maintaining or improving performance.
By Ivan Anokhin, Johan Obando-Ceron, Irina Rish, Sebastian Risi
The Asymmetric Looped Transformer (AsymLoop) separates context encoding from state refinement by using a single-pass causal encoder to produce fixed global key-value caches and a weight-tied recurrent decoder to generate local key-values from its evolving state. Joint attention over both sources allows each update to combine persistent contextual information with the current intermediate state. Experiments on FineWeb-Edu across models from 60M to 1B show lower validation perplexity and higher downstream accuracy than full-loop models, with a 1B model improving zero-shot commonsense reasoning accuracy from 49.58% to 50.85%.
By Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang, Zhouhan Lin