← Back to all news
arXiv Computation and Language

Quantizing Recurrent Language Models Is Easier Than You Think

The paper investigates how to quantize recurrent language models, which use linear recurrences instead of softmax attention. It distinguishes between extra‑state quantization (weights and activations outside the recurrent state) and intra‑state quantization (the recurrent state itself during decoding). The authors introduce LoReS, a quantization pipeline that applies value‑axis scale grouping, decay‑based mixed precision, and stochastic rounding, enabling robust 8‑bit state quantization for large models such as Qwen‑3.5, Nemotron‑H, and Kimi‑Linear, and achieving decode speedups when integrated into vLLM.

Related stories

arXiv Machine Learning
Aug 31

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

The paper introduces DAMP, a decay‑aware mixed‑precision quantization scheme for recurrent‑state representations in GDN and KDA language models. By identifying high‑risk channels through quantization‑error energy and decay persistence, DAMP stores these channels at higher precision while compressing the rest to INT8, achieving a 9.9‑bit average precision. Experiments on Qwen3.6‑35B and Kimi‑Linear‑48B show a 69.1% reduction in recurrent‑state storage, up to 2.01× faster state‑update kernels, and up to 10.9% lower full‑model TPOT while preserving accuracy close to the FP32 baseline.

By Tao Zhang, Jianchao Tan, Pingwei Sun, Yanqi Yu, Zixu Jiang, Yuchen Xie, Xunliang Cai, Ziqian Zeng
llmsefficiencybenchmarks
More like this →
arXiv AI
Sep 30

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

arXiv:2609.38166v1 Announce Type: cross Abstract: Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Atte...

By Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra, Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer, Rishabh Iyer, Ion Stoica
llmsefficiency
More like this →
Hugging Face Trending Papers
Jun 3

STaR-Quant: State-Time Consistent Post-Training Quantization for Diffusion Large Language Models

Diffusion large language models (DLLMs) have recently emerged as a promising alternative to autoregressive LLMs by generating text through iterative masked denoising with bidirectional context. However, their large model sizes and iterative denoising process introduce substantial memory and computational overhead, motivating post-training quantization for efficient deployment.

llmsdiffusionefficiency
More like this →
arXiv Machine Learning
Aug 27

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi
llmsefficiency
More like this →