arXiv Computation and Language

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

arXiv AI
Oct 7

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration

The paper introduces Quantizer‑Aligned Recalibration (QuAR), a single‑pass test‑time adaptation technique for quantized vision transformers that does not require backpropagation or parameter updates. QuAR recalibrates activations at the input of frozen quantizers by aligning per‑channel statistics with the source calibration, thereby correcting the distorted code distribution caused by distribution shift. On ImageNet‑C, QuAR outperforms state‑of‑the‑art backprop‑free methods across 3‑, 4‑, 6‑, and 8‑bit precisions, achieving higher accuracy, lower latency, and minimal memory overhead while maintaining performance across diverse shift scenarios.

By Hyeongheon Cha, Young D. Kwon, Sung-Ju Lee
arXiv Computation and Language
Nov 4

Quantizing Recurrent Language Models Is Easier Than You Think

The paper investigates how to quantize recurrent language models, which use linear recurrences instead of softmax attention. It distinguishes between extra‑state quantization (weights and activations outside the recurrent state) and intra‑state quantization (the recurrent state itself during decoding). The authors introduce LoReS, a quantization pipeline that applies value‑axis scale grouping, decay‑based mixed precision, and stochastic rounding, enabling robust 8‑bit state quantization for large models such as Qwen‑3.5, Nemotron‑H, and Kimi‑Linear, and achieving decode speedups when integrated into vLLM.

By Philipp Nazari, Neehal Tumma, Daniela Rus, T. Konstantin Rusch
arXiv Machine Learning
Oct 12

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers

The paper introduces TTS‑Q, a perception‑guided post‑training quantization method for zero‑shot text‑to‑speech diffusion transformers. It identifies that 4‑bit weight‑and‑activation quantization erodes speaker timbre, and addresses this by selecting hard‑to‑quantize items and applying a mixed‑precision schedule that gives higher precision to late‑step feed‑forward activations. The resulting approach restores timbre while preserving intelligibility and reduces GPU memory usage for resource‑constrained deployment.

By Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen
arXiv Machine Learning
Oct 8

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.

By Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim
arXiv Machine Learning
Oct 2

QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs

arXiv:2609.39223v2 Announce Type: new Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...

By Weili Xu, Jisen Li, Yuqing Jian, Chenxi Li, Zhizhou Sha, Yifan Yu, Qingyang Wu, Chenfeng Xu, Zhongzhu Zhou, Tianyi Zhang, Ben Athiwaratkun