arXiv AI
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
1d ago
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...
arXiv AI
Oct 5
Tailoring the Quantization Space for 1-Bit KV Cache Compression
arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
arXiv Machine Learning
Jul 2
GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
arXiv Machine Learning
Sep 1
SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
arXiv:2608.28911v1 Announce Type: new Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length....
arXiv Machine Learning
1d ago
PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
PN-Mix is a mixed‑precision key‑cache quantization technique for RoPE‑based decoder‑only large language models that allocates higher precision to salient channel pairs identified during the prefill stage. By focusing on channel pairs with large query–key pair‑norm products, PN‑Mix reduces logit variation caused by quantization errors. Experiments on reasoning, code‑generation, and AIME benchmarks show that PN‑Mix achieves comparable or better accuracy than prior methods while using an average precision of 2.26 bits and degrades gracefully down to 1.5 bits.
arXiv AI
Aug 6
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.