arXiv Computation and Language
StateLoop: Persistent State for Looped Transformers
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
6d ago
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
arXiv:2610.35556v1 Announce Type: new Abstract: Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream...
arXiv Machine Learning
Sep 1
You Do Not Fully Utilize Transformer's Representation Capacity
arXiv:2502.09245v3 Announce Type: replace Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...
arXiv Machine Learning
Oct 3
Decoding Looped Transformers Better for (Almost) Free
The paper introduces LoopCD, a training‑free contrastive decoding framework that improves token selection in Loop‑Transformer models by comparing the final prediction with earlier recurrent passes. LoopCD operates either in logit space (LoopCD‑Logits) with a single extra output pass or in hidden‑state space (LoopCD‑Hidden) with no output overhead. Across multiple looped Transformer families, LoopCD yields significant performance gains—raising pass@1 scores on tasks such as AIME 2024 and HumanEval—while enabling a reduction in the number of recurrent loops and a corresponding decrease in inference FLOPs.
arXiv Machine Learning
Sep 16
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
arXiv:2609.17184v1 Announce Type: new Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurren...
arXiv Machine Learning
Sep 25
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
arXiv AI
Oct 16
LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models
LoopAhead is a speculative decoding framework designed for looped Transformers, which repeatedly apply the same Transformer layers to improve parameter efficiency. It introduces a lightweight lookahead drafter (LAD) that predicts multiple future tokens using early-loop states, while deeper-loop feedback determines which draft branches to continue or pause, and the full-depth model finalizes token decisions. This approach achieves up to 7.82× faster decoding than traditional autoregressive methods while maintaining the target model’s output distribution.