arXiv Machine Learning
Looped Transformers as Optimizers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 1
Sparse Layers are Critical to Scaling Looped Language Models
arXiv:2605. 09165v2 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries.
arXiv AI
Jul 14
Looped State-Space Language Models with Adaptive Exit-State Selection
arXiv:2607. 10110v1 Announce Type: new Abstract: Recent work on looped language models suggests that many reasoning problems benefit from greater computational depth rather than from additional independent parameters.
arXiv Machine Learning
6d ago
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
arXiv AI
1d ago
LoopICL: Looping a single transformer block to solve tabular tasks
LoopICL is a transformer architecture that loops a single block to address tabular tasks. It separates parameter count from computational depth by using a cell stream for per‑cell features and a row stream for in‑context examples, refined via within‑column and cross‑column attention. During pre‑training, varying loop counts and a learned exit‑gate allow the model to adjust inference depth at test time, achieving competitive performance with TabICLv2 while using about 90% fewer parameters.
arXiv Machine Learning
Sep 16
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
arXiv:2609.17184v1 Announce Type: new Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurren...
arXiv AI
Jun 4
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
arXiv:2606. 04438v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth.