arXiv Computation and Language
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Oct 3
Decoding Looped Transformers Better for (Almost) Free
The paper introduces LoopCD, a training‑free contrastive decoding framework that improves token selection in Loop‑Transformer models by comparing the final prediction with earlier recurrent passes. LoopCD operates either in logit space (LoopCD‑Logits) with a single extra output pass or in hidden‑state space (LoopCD‑Hidden) with no output overhead. Across multiple looped Transformer families, LoopCD yields significant performance gains—raising pass@1 scores on tasks such as AIME 2024 and HumanEval—while enabling a reduction in the number of recurrent loops and a corresponding decrease in inference FLOPs.
arXiv AI
Oct 16
LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models
LoopAhead is a speculative decoding framework designed for looped Transformers, which repeatedly apply the same Transformer layers to improve parameter efficiency. It introduces a lightweight lookahead drafter (LAD) that predicts multiple future tokens using early-loop states, while deeper-loop feedback determines which draft branches to continue or pause, and the full-depth model finalizes token decisions. This approach achieves up to 7.82× faster decoding than traditional autoregressive methods while maintaining the target model’s output distribution.
arXiv Machine Learning
Sep 16
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
arXiv:2609.17184v1 Announce Type: new Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurren...
arXiv Machine Learning
Sep 2
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
arXiv:2605.26797v2 Announce Type: replace Abstract: We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden...
arXiv AI
2d ago
CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening
CopyLoop is a training‑free decoding algorithm for looped transformers that speeds up generation by proposing multi‑token suffixes from repeated context, evaluating them early at a shallow recurrent depth, and scheduling continuous recurrent processing. It avoids step‑by‑step decoding, allowing newer blocks to be processed concurrently while earlier blocks continue toward full‑depth verification. Across six model checkpoints from the Ouro, Raven, and Huginn families, CopyLoop delivers 2.98x–10.70x speedups on Base models and 2.74x–3.52x on Thinking models, outperforming or matching all competing baselines on code generation and multi‑step mathematical reasoning tasks.
arXiv Computation and Language
6d ago
StateLoop: Persistent State for Looped Transformers
arXiv:2610.35544v1 Announce Type: new Abstract: Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use...