arXiv AI
Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 5
Tailoring the Quantization Space for 1-Bit KV Cache Compression
arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
arXiv Machine Learning
21h ago
StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference
arXiv:2611.08363v1 Announce Type: new Abstract: Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision dat...
arXiv AI
6d ago
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
The paper introduces GAAC, a lossless key‑value (KV) compression method for large language model inference on personal devices. GAAC encodes BF16 exponents with per‑channel base‑plus‑offset codes, adapts code widths across small groups, and reconstructs KV tiles on the fly to avoid full‑cache decompression. Experiments show 1.4× compression for BF16 KV, speedups of up to 1.27× in decode attention, and a 26% reduction in resident KV memory while boosting throughput on NVIDIA and Apple hardware.
arXiv Machine Learning
Sep 29
The KV Cache Is the New Memory Wall
The paper argues that for large‑context autoregressive language‑model inference, memory bandwidth—specifically the Key‑Value (KV) cache—becomes the limiting resource rather than arithmetic throughput. It analytically derives how arithmetic intensity decays with context length for NVIDIA H100, NVIDIA B200, and AMD MI300X, identifies crossover points where KV traffic overtakes weight traffic, and evaluates representative techniques across five compression domains. The study finds a three‑regime behavior: below the crossover, weight traffic dominates and KV compression offers little benefit; beyond it, KV traffic dominates and compression methods trade quality for bandwidth, with paging and prefix sharing being lossless but capacity‑limited, while quantization and eviction directly reduce bandwidth at the cost of accuracy. whyItMatters":"The work provides a unified analytical framework and a standardized protocol that enable consistent comparison of KV‑compression techniques across hardware and workloads, guiding practitioners in selecting appropriate methods for long‑context inference."
arXiv Machine Learning
Jun 24
RoPE-Aware Bit Allocation for KV-Cache Quantization
arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.
arXiv Machine Learning
Jun 29
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.