arXiv AI
Aug 12
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.
Best-of-Depth: Learning Exploratory Dynamics in Looped Transformers
Related stories
arXiv Computation and Language
Sep 30
Pretraining Latent Information Feedback Transformers with Teacher Supervision
arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...
arXiv Computation and Language
Oct 26
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
arXiv:2610.32492v1 Announce Type: cross Abstract: Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substant...
arXiv Machine Learning
Aug 18
RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers
arXiv:2608. 15062v1 Announce Type: cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency.
arXiv AI
Sep 16
Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
arXiv:2603.21676v2 Announce Type: replace-cross Abstract: Standard Transformers have a fixed computational depth, limiting their ability to generalize to tasks that require variable-depth reasoning....
arXiv Machine Learning
Sep 30
What Makes Recurrence Effective in Looped Language Models?
arXiv:2609.36636v1 Announce Type: new Abstract: Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding...
arXiv Machine Learning
Jun 8
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
arXiv:2606. 06574v1 Announce Type: new Abstract: Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers.
arXiv Machine Learning
Oct 9
One Block, Multiple Depths: Recurrent Vision Transformers with Depth-Programmed Experts
The paper introduces reViT, a recurrent vision transformer that uses a single Transformer block repeatedly to achieve the accuracy of a full-depth encoder while keeping inference FLOPs comparable. It achieves depth‑specific transformations by expressing each block’s feed‑forward network as a convex combination of a small shared expert bank, with a continuous depth coordinate guiding the mixture. Experiments on ImageNet‑1k and DINOv2 distillation show that reViT matches or surpasses state‑of‑the‑art models, reduces stored parameters by about 70%, and supports elastic‑depth deployment from a single checkpoint.
arXiv Computation and Language
Oct 27
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
arXiv:2610.35556v1 Announce Type: new Abstract: Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream...
arXiv AI
Jun 2
Agentic Transformers Provably Learn to Search via Reinforcement Learning
arXiv:2606. 00183v1 Announce Type: cross Abstract: Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks: agents must explore actions, remember failures, and backtrack toward promising alternatives.
arXiv AI
Aug 19
LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models
LoopVLA introduces a recurrent Vision‑Language‑Action architecture that learns to refine multimodal representations, predict actions, and estimate when further refinement is unnecessary. By iteratively applying a shared Transformer block and producing a sufficiency score at each step, it decouples refinement from fixed layer indices and aligns confidence scores with action quality through a self‑supervised objective. Experiments on LIBERO, LIBERO‑Plus, and VLA‑Arena demonstrate that LoopVLA reduces model parameters by 45% and boosts inference throughput up to 1.7× while matching or surpassing strong baselines in task success.
arXiv Machine Learning
Oct 9
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
The State Stream Transformer (SST) V2 introduces a parameter‑efficient, FFN‑driven nonlinear recurrence that streams latent states horizontally across a sequence, enabling continuous latent deliberation at inference time. A two‑pass parallel training procedure approximates this recurrence, making co‑training practical. Experiments on GSM8K and GPQA‑Diamond show that SST achieves a 61.11% architectural capacity bound and a 15.15‑point improvement over a fine‑tuning‑matched baseline, while reducing GSM8K errors by 46%.