← Back to all news
arXiv Machine Learning By Tejinder Singh

The KV Cache Is the New Memory Wall

Read the original on arXiv Machine Learning →

The paper argues that for large‑context autoregressive language‑model inference, memory bandwidth—specifically the Key‑Value (KV) cache—becomes the limiting resource rather than arithmetic throughput. It analytically derives how arithmetic intensity decays with context length for NVIDIA H100, NVIDIA B200, and AMD MI300X, identifies crossover points where KV traffic overtakes weight traffic, and evaluates representative techniques across five compression domains. The study finds a three‑regime behavior: below the crossover, weight traffic dominates and KV compression offers little benefit; beyond it, KV traffic dominates and compression methods trade quality for bandwidth, with paging and prefix sharing being lossless but capacity‑limited, while quantization and eviction directly reduce bandwidth at the cost of accuracy. whyItMatters":"The work provides a unified analytical framework and a standardized protocol that enable consistent comparison of KV‑compression techniques across hardware and workloads, guiding practitioners in selecting appropriate methods for long‑context inference."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 20

Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.

By Shriniwas Ramesh Suram
llms
More like this →
arXiv Machine Learning
Sep 11

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

The paper presents py‑kvcache, a new KV offload connector for vLLM that uses asynchronous direct I/O, bounded shared staging, and scheduler‑aware preloading to improve external KV caching performance on NVMe SSDs. Experiments across synthetic workloads, long‑context benchmarks, and production traces show that py‑kvcache can load 80k‑token prefixes 2.0× faster than LMCache, with preloading contributing an additional 1.34× speedup, and achieves overall performance within 4% of native vLLM KV Offload. The study highlights that cache effectiveness depends on transfer granularity, intermediate memory use, and scheduling timing rather than just device bandwidth, indicating that external KV caching should be considered a setup‑specific admission decision.

By Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 18

PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

PrefixBench-H100 is a reproducible benchmark that evaluates how reusing prompt prefixes affects LLM serving performance on NVIDIA H100 GPUs. It tests two popular runtimes (vLLM and TensorRT-LLM) across varied workloads, measuring metrics such as time-to-first-token, latency, throughput, cache hits, and GPU memory usage. The study identifies when prefix reuse significantly reduces first‑token latency and when cache pressure diminishes those gains, noting that cache effectiveness is largely unaffected by concurrency or output length, while differences arise mainly in scheduling.

By Omkar Shewale, Deepak Kumar, Divakar Kumar Yadav
llmsragagentsefficiencybenchmarks
More like this →