arXiv AI
Aug 11
Full-bandwidth transformer
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute
UpLoop introduces a pretraining schedule that trains looped Transformers by upcycling early training computations, allowing the model to increase execution depth without adding extra parameters or training FLOPs. The method starts with a shallow Transformer, progressively copies learned layers to deepen the network, and reuses saved computation to support repeated execution of a shared middle block. Experiments across various model sizes demonstrate that UpLoop improves language modeling and downstream task performance, raising the average score across 22 tasks from 22.99 to 27.52 for a 1.3B‑parameter model, and also benefits mathematical reasoning after fine‑tuning.
Related stories
arXiv AI
6d ago
How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
The paper investigates how looping—reusing transformer weights to deepen models without adding parameters—affects performance under different constraints. Experiments on dense decoder‑only transformers up to 1 B parameters show that looping the full transformer yields best quality for a fixed parameter budget, while looping a middle block is preferable when training FLOPs are limited. Adding loop‑specific low‑rank adapters improves quality, but decoding slows linearly with loops; a parallel loop design that passes hidden states across token positions can mitigate this slowdown, though careful handling of token embeddings is needed to retain gains.
arXiv Computation and Language
Oct 27
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
arXiv:2610.35556v1 Announce Type: new Abstract: Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream...
arXiv Machine Learning
Sep 25
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
arXiv Machine Learning
Sep 30
Looped Transformers as Optimizers
arXiv:2609.37379v1 Announce Type: new Abstract: Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models h...
arXiv Computation and Language
Oct 26
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
arXiv:2610.32492v1 Announce Type: cross Abstract: Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substant...
arXiv AI
Oct 2
Closing the Loop: Practical Training Recipes for Looped Language Models
The paper presents practical training recipes for looped language models, showing that a compute‑efficient pipeline can reduce the training budget from 7.7 T tokens to 310 B tokens while maintaining strong reasoning performance. In controlled experiments, a 1.4 B LoopLM outperforms a parameter‑matched dense model on 12 benchmarks, achieving significant gains on GSM8K, MATH, and DROP, and approaches a 3.9 B dense model at matched inference compute. The authors also provide a minimal recipe to convert pretrained dense models into looped ones, demonstrating improvements on Qwen3‑1.7B‑Base across multiple data regimes.
arXiv AI
6d ago
CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening
CopyLoop is a training‑free decoding algorithm for looped transformers that speeds up generation by proposing multi‑token suffixes from repeated context, evaluating them early at a shallow recurrent depth, and scheduling continuous recurrent processing. It avoids step‑by‑step decoding, allowing newer blocks to be processed concurrently while earlier blocks continue toward full‑depth verification. Across six model checkpoints from the Ouro, Raven, and Huginn families, CopyLoop delivers 2.98x–10.70x speedups on Base models and 2.74x–3.52x on Thinking models, outperforming or matching all competing baselines on code generation and multi‑step mathematical reasoning tasks.
arXiv Computation and Language
Oct 27
StateLoop: Persistent State for Looped Transformers
arXiv:2610.35544v1 Announce Type: new Abstract: Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use...
arXiv Machine Learning
Aug 31
Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning
arXiv:2511.08577v4 Announce Type: replace-cross Abstract: Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applicat...
arXiv Machine Learning
Jul 1
Sparse Layers are Critical to Scaling Looped Language Models
arXiv:2605. 09165v2 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries.
arXiv Machine Learning
Sep 2
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
arXiv:2609.01343v1 Announce Type: new Abstract: Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating archit...