← Back to all news
arXiv Computation and Language By Philipp Nazari, Neehal Tumma, Daniela Rus, T. Konstantin Rusch

Quantizing Recurrent Language Models Is Easier Than You Think

Read the original on arXiv Computation and Language →

The paper investigates how to quantize recurrent language models, which use linear recurrences instead of softmax attention. It distinguishes between extra‑state quantization (weights and activations outside the recurrent state) and intra‑state quantization (the recurrent state itself during decoding). The authors introduce LoReS, a quantization pipeline that applies value‑axis scale grouping, decay‑based mixed precision, and stochastic rounding, enabling robust 8‑bit state quantization for large models such as Qwen‑3.5, Nemotron‑H, and Kimi‑Linear, and achieving decode speedups when integrated into vLLM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 31

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

The paper introduces DAMP, a decay‑aware mixed‑precision quantization scheme for recurrent‑state representations in GDN and KDA language models. By identifying high‑risk channels through quantization‑error energy and decay persistence, DAMP stores these channels at higher precision while compressing the rest to INT8, achieving a 9.9‑bit average precision. Experiments on Qwen3.6‑35B and Kimi‑Linear‑48B show a 69.1% reduction in recurrent‑state storage, up to 2.01× faster state‑update kernels, and up to 10.9% lower full‑model TPOT while preserving accuracy close to the FP32 baseline.

By Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng
llmsefficiencybenchmarks
More like this →
arXiv AI
Sep 30

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

arXiv:2609.38166v1 Announce Type: cross Abstract: Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Atte...

By Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica
llmsefficiency
More like this →