The Asymmetric Looped Transformer (AsymLoop) separates context encoding from state refinement by using a single-pass causal encoder to produce fixed global key-value caches and a weight-tied recurrent decoder to generate local key-values from its evolving state. Joint attention over both sources allows each update to combine persistent contextual information with the current intermediate state. Experiments on FineWeb-Edu across models from 60M to 1B show lower validation perplexity and higher downstream accuracy than full-loop models, with a 1B model improving zero-shot commonsense reasoning accuracy from 49.58% to 50.85%.
By Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang, Zhouhan Lin
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
By Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin, Inessa Fedorova, Dmitry Bocharov, Yuliana Shakhvalieva, Maria Tikhonova, Valerii Ternovskii
The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.
By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
arXiv:2606. 04438v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth.
By Wenkai Chen, Tianshu Li, Wenyong Huang, Yichun Yin, Lifeng Shang, Chengwei Qin
arXiv:2607. 10110v1 Announce Type: new Abstract: Recent work on looped language models suggests that many reasoning problems benefit from greater computational depth rather than from additional independent parameters.
By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
arXiv:2606. 27538v1 Announce Type: cross Abstract: We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block that pre-contextualizes each token before it enters the block.
By Mahesh Godavarti