← Back to all news
arXiv Machine Learning By Shiyu Ji, Peng Ling, Chen Lin, Shupeng Li, Yixuan Wang, Qingfu Zhu, Wanxiang Che

RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention

Read the original on arXiv Machine Learning →

RoLaQ is an offline technique that jointly calibrates rotation and quantization range for 2‑bit key‑value caches in multi‑head latent attention models. By refining a shared rotation within subspaces defined by frozen key and value projections and learning coupled quantization thresholds and reconstruction gains, RoLaQ improves prediction matching and attention‑output supervision while keeping model weights frozen. Experiments on three MLA models and six benchmarks show that RoLaQ outperforms other 2‑bit methods, achieving higher average accuracy and, in a DeepSeek‑V2‑Lite implementation, nearly four times the BF16 KV capacity and more than double the decode throughput for 256K input tokens.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
6d ago

RAQ-MLA: Role-Aware Quantization for Multi-Head Latent Attention Caches

RAQ-MLA introduces a role-aware quantization scheme for Multi-Head Latent Attention (MLA) caches used in large language models. It analyzes the distinct functions of the cached latent and positional key tensors, applying tailored invertible transforms to each based on their impact on attention output. The method achieves near-lossless performance while reducing the cache size to about 2.5 bits per element, effectively increasing theoretical KV-cache capacity by 6.5×.

By Tianqi Chen, Xunchao Li, Linchuan Xie, Guanghua Yu, Jianchen Zhu, Peisong Wang, Jian Cheng
llmsefficiency
More like this →
arXiv Machine Learning
Jul 1

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Jun 9

SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference

arXiv:2605. 18856v3 Announce Type: replace Abstract: Long-context inference is increasingly constrained by the KV cache: resident memory grows with context length, and decoding becomes limited by repeated High Bandwidth Memory (HBM) streaming rather than arithmetic.

By Anay Chauhan, Gurucharan Marthi Krishna Kumar, Arion Das, Amit Dhanda, Vinija Jain, Aman Chadha, Amitava Das
efficiency
More like this →