ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.
By Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim
arXiv:2610.16368v1 Announce Type: cross
Abstract: Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over...
By Zhixiong Zhao, Zukang Xu, Xing Hu, Lifeng Liu, Guangyu Sun, Dawei Yang
Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match...
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford
The paper investigates how looping—reusing transformer weights to deepen models without adding parameters—affects performance under different constraints. Experiments on dense decoder‑only transformers up to 1 B parameters show that looping the full transformer yields best quality for a fixed parameter budget, while looping a middle block is preferable when training FLOPs are limited. Adding loop‑specific low‑rank adapters improves quality, but decoding slows linearly with loops; a parallel loop design that passes hidden states across token positions can mitigate this slowdown, though careful handling of token embeddings is needed to retain gains.
By Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla
The paper introduces TRACE, a framework that embeds a Vector Quantizer at the post‑attention residual of each Transformer layer, enabling exchanged activations to be encoded as compact codebook indices. During inference, receivers recover key/value representations via table lookup, eliminating downstream computation and allowing communication–computation overlap. A hierarchical codebook further permits runtime bitrate adaptation, and experiments on ViT‑Base and ViViT‑B demonstrate up to 1.89% accuracy gain, 1.27× speedup, and a 22,624× reduction in communication overhead compared to uncompressed inference.
By Muhammad Azlan Qazi, Alexandros Iosifidis, Qi Zhang