arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford
The paper investigates how looping—reusing transformer weights to deepen models without adding parameters—affects performance under different constraints. Experiments on dense decoder‑only transformers up to 1 B parameters show that looping the full transformer yields best quality for a fixed parameter budget, while looping a middle block is preferable when training FLOPs are limited. Adding loop‑specific low‑rank adapters improves quality, but decoding slows linearly with loops; a parallel loop design that passes hidden states across token positions can mitigate this slowdown, though careful handling of token embeddings is needed to retain gains.
By Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla
arXiv:2610.35556v1 Announce Type: new
Abstract: Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream...
By Feichen Song, Boyi Zeng, Yiqin Hao, Shixiang Song, He Li, Yizhou Yang, Zitong Wang, Yifan Liu, Ziwei He, Zhouhan Lin
FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
By Wanqi Yang, Shiwei Liu
arXiv:2609.37379v1 Announce Type: new
Abstract: Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models h...
By Yulong Huang, Chen Jiang, Zhanpeng Zhou, Hongtao Zhang, Tianyu Li, Tianyu He, Xiangyu Zhang, Bojun Cheng
arXiv:2610.32492v1 Announce Type: cross
Abstract: Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substant...
By Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang