arXiv:2609.33149v2 Announce Type: replace
Abstract: A common principle of effective learning is to practice material that is neither already mastered nor too difficult to permit progress. We ask how...
By Hongbo Chen, Guohua Lu, Ting Dang, Hong Jia
The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.
By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
TraceRelay is an attention‑aligned recurrent architecture that distributes persistent representations over a rolling sequence of low‑dimensional traces. It uses right‑looking attention to form increments from lower‑layer representations, delays delivery until all attended inputs are in the causal past, and accumulates these increments with a fixed additive phase recurrence. Experiments on 36 small‑model runs show that recurrent phase inheritance yields high accuracy on Equal Repeats (up to 99.69%) but accuracy drops beyond the training range, and that the optimal size of recurrent traces varies across tasks such as Dyck closing‑type prediction and causal Most‑Freq generation.
By Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
The paper introduces Elastic Latent Recurrence (ELR), a training recipe that enables looped language models to match the accuracy of non-recurrent models at a single loop while allowing additional loops to improve accuracy with more compute. ELR achieves this through gated state updates with learned step sizes, mixed loop counts during training, and conditioning on loop index and budget. The method also reduces training and inference overhead in Mixture-of-Experts models and produces a single checkpoint, Koi‑4B, that performs competitively with state‑of‑the‑art models on multiple benchmarks.
By Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada
arXiv:2610.19768v1 Announce Type: new
Abstract: Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known...
By Peilong Zhou, Hongtao Zhang, Yulong Huang, Cangyuan Li, Mingyu Chen, Ying Wang
arXiv:2607. 14427v1 Announce Type: new Abstract: A depth-recurrent transformer applies a weight-tied core a variable number of times, and prior work has shown that training with a randomized recursion count yields one checkpoint usable across a range of inference depths.
By Joe Logan