FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.
By Wanqi Yang, Shiwei Liu
ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.
By Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim
arXiv:2601. 22580v2 Announce Type: replace-cross Abstract: The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures.
By Chao Wang, Bei Li, Jiaqi Zhang, Xinyu Liu, Yuchun Fan, Linkun Lyu, Xin Chen, Jingang Wang, Tong Xiao, Peng Pei, Xunliang Cai
The paper presents TTS‑Q, a post‑training quantization method that reduces the size of large diffusion transformer TTS models to 4‑bit precision while preserving speaker timbre and intelligibility. By identifying hard‑to‑quantize weights and activations and applying a mixed‑precision schedule that gives higher precision to late‑step feed‑forward activations, TTS‑Q restores timbre loss observed in standard 4‑bit PTQ. The approach also offers a calibrated‑W4A4 reference that lowers GPU memory usage, making TTS diffusion transformers more viable for on‑device, assistive, and real‑time applications.
By Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
The paper introduces TTS‑Q, a perception‑guided post‑training quantization method for zero‑shot text‑to‑speech diffusion transformers. It identifies that 4‑bit weight‑and‑activation quantization erodes speaker timbre, and addresses this by selecting hard‑to‑quantize items and applying a mixed‑precision schedule that gives higher precision to late‑step feed‑forward activations. The resulting approach restores timbre while preserving intelligibility and reduces GPU memory usage for resource‑constrained deployment.
By Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
InfoQuant is a train‑free method that reshapes activation distributions to be more compatible with low‑bit quantizers. It uses Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier‑token selection to reduce numerical range while maintaining dispersion, thereby lowering quantization error. Across several large language model families, InfoQuant outperforms existing post‑training quantization and end‑to‑end training baselines, preserving 97% of floating‑point accuracy on average and narrowing the LLaMA‑2 13B performance gap by 42%.
By Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang
arXiv:2607. 10137v1 Announce Type: new Abstract: Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision.
By Prateek Singh
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford
arXiv:2609.30820v1 Announce Type: cross
Abstract: Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes...
By Nux Li
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
arXiv:2610.07522v1 Announce Type: new
Abstract: Post-training quantization is a powerful tool for compressing large language models. The most scalable methods quantize every layer in parallel, but qu...
By Yan Scholten, Rachel Lawrence, James Hensman, Stephan G\"unnemann, Alicia Curth, Riccardo Grazzi
arXiv:2506. 04985v2 Announce Type: replace Abstract: Large language models (LLMs) require substantial compute, and thus energy, at inference time.
By Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel