VestigeKV is a new KV‑cache technique that uses a 64‑dimensional vestigial branch—originally a RoPE component repurposed during NoPE training—as a query‑independent eviction signal. By reading only 11 % of each cache row, the method partitions the cache into an attended tier (top‑m rows) and an archive tier (all other rows), which is GPU‑resident and never deleted. The approach achieves near‑perfect retrieval (1.00 at 8×, 0.92 at 32×) without any training, quantization, or changes to weights or kernels.
By WenJie Fan
arXiv:2606. 01502v1 Announce Type: cross Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk.
By Bole Ma, Jan Eitzinger, Harald K\"ostler, Gerhard Wellein
arXiv:2606. 09916v1 Announce Type: cross Abstract: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.
By Junjie Li, Jiong Lou, Jie Li
BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.
By Jiamu Zhang, Hanjie Chen
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
By Haeyong Kang, Chang D. Yoo
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
By Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
By Sergii Kozyrev (Minima AI, Inc), Davyd Maiboroda (Minima AI, Inc)
The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %.
whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."
By Muhammad Ibrahim
The paper introduces a method to incorporate attention bias directly into the keys of compacted KV caches, enabling standard kernels like FlashAttention and vLLM to use these folded caches without modification. By fitting a bias along a closed‑form direction per head, the approach preserves the cache size and head dimension while bounding output error based on query projections. Experiments on Qwen3 and Llama‑3.1 models show that the folded caches match or exceed the accuracy of the original Attention Matching method across various benchmarks, with comparable or lower KL divergence to full caches.
By Utkarsh Kanwat
The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.
By Fanxu Meng, Zhixin Pan
arXiv:2608. 11231v1 Announce Type: new Abstract: LLM serving is increasingly accelerated by position-independent caching (PIC).
By Yirui Liu, Ruoling Qi, Longwen Wang, Xuaner Wu, Jian Chen, Yuxin Jin, Jiawei Shao, Xuelong Li