arXiv Machine Learning

What Makes Recurrence Effective in Looped Language Models?

arXiv AI
Sep 3

CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models

The paper introduces CHASE, a cache‑hole‑adapted skip‑exit mechanism for looped state‑space language models, specifically Looped Mamba and Looped Hybrid Mamba‑Transformer. It shows that looping these architectures improves performance on controlled reasoning tasks and remains competitive in pre‑training benchmarks while using fewer distinct parameters. The cache‑hole adaptation allows selective skipping of recurrent steps during inference, maintaining perplexity close to full computation and achieving significant speedups.

By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa
arXiv AI
Oct 2

Closing the Loop: Practical Training Recipes for Looped Language Models

The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.

By Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin, Inessa Fedorova, Dmitry Bocharov, Yuliana Shakhvalieva, Maria Tikhonova, Valerii Ternovskii
arXiv AI
Oct 9

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

The paper introduces InfiLoop, a loop-native residual connection designed for looped Transformers to prevent performance degradation as iterations increase. InfiLoop learns which past computations to retain and how much to accept from each new update, combining content-based weighting with learned temporal decay to maintain a running summary of recurrent states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, achieving 97.9% exact accuracy on Sudoku-Extreme and 13.6% pass@2 on ARC-AGI-2, and continues to improve with test-time looping beyond 20,000 effective steps.

By Pengxiang Li, Dilxat Muhtar, Di He, Guinan Su, Lu Yin, Shiwei Liu
arXiv Machine Learning
Aug 19

Recirculation

The paper introduces recirculation, an inference‑time architectural enhancement for foundation models that reduces perplexity and improves accuracy on generation and reasoning tasks without adding significant latency. Recirculation adds a specific form of recurrence, enabling the model to function as a dynamical system that tracks belief states, and is distinct from chain‑of‑thought or depth‑recurrence methods. An adaptive variant requires minimal hyperparameter tuning and achieves notable gains on the Gemma3 family, including a 23% perplexity drop and a 21% accuracy increase on GSM8k.

By Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu
arXiv AI
Aug 20

Looped Language Models Improve Compositional Tool Calling

The paper investigates the use of looped language models for compositional tool calling, where models must coordinate multiple API calls and maintain state across interactions. Experiments on API-Bank, BFCL, and NESTful show that recurrent computation generally improves compositional and dependency-aware tool use, with accuracy increasing as recurrent depth grows. Adaptive inference offers a better compute‑performance trade‑off by allocating extra computation only when necessary.

By Andrei Cristian Popescu, Haitz S\'aez de Oc\'ariz Borde, Pietro Li\`o
arXiv Machine Learning
Sep 4

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

RecurTrace introduces adaptive latent reasoning for language models by allowing each looped layer to attend to its own past states and by using a halting head to decide when to stop iterating. This approach overcomes two limitations of prior latent recurrence methods: limited access to earlier computations and a fixed loop count that mismatches input difficulty. In experiments on MathQA, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, outperforming fixed‑depth baselines and other adaptive methods, and it also improves generation accuracy across a range of model sizes.

By Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu, Junyu Wang, Zhizheng Wu
Hugging Face Trending Papers
Oct 8

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

The paper proposes InfiLoop, a loop-native residual connection designed for looped Transformers to prevent performance degradation as iterations increase. InfiLoop learns which past computations to retain and how much to accept from each new update, using content-based weighting and learned temporal decay to maintain a running summary of recurrent states. The approach yields strong results on reasoning tasks, achieving 97.9% exact accuracy on Sudoku-Extreme and 13.6% pass@2 on ARC-AGI-2, and continues to improve with over 20,000 test-time iterations.

arXiv Machine Learning
Oct 9

Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

LoopOPD introduces a cross-loop on-policy distillation framework that uses a frozen terminal loop policy as a compute-privileged teacher for intermediate loop students, providing dense supervision without external teachers. Dynamic LoopOPD (D-LoopOPD) further refreshes the terminal loop teacher as model parameters update, enabling recurrent self‑improvement. Experiments on Ouro‑Thinking models show that LoopOPD enhances mathematical reasoning and that D‑LoopOPD yields additional gains, with the models also improving on general reasoning and code generation benchmarks.

By Yi Wang, Rui Qian, Yu Li, Haoyang Yao, Wenjie Wang
arXiv Computer Vision
Oct 23

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models

LoopLook introduces a lightweight encoder-side recurrent perception framework that equips pretrained Vision‑Language Models with adaptive visual depth. By repeatedly revisiting a middle‑to‑late window of the vision encoder and injecting question representations at each loop, it refines visual evidence in a task‑directed manner. The learned stopping mechanism and token‑wise Mixture‑of‑Looks allow each visual token to selectively integrate information across loops, yielding consistent performance gains across multiple backbones and benchmarks.

By Wencheng Ye, Anning Hu, Hongye Fang, Tianyi Wang, Junchi Yan