EntropyCache is a training‑free KV caching technique for diffusion‑based large language models that uses the maximum entropy of newly decoded token distributions to decide when to recompute cached states. By leveraging two empirical observations—entropy correlating with cache drift and feature volatility persisting across multiple steps—EntropyCache makes a skip‑or‑recompute decision with only O(V) computation per step, independent of context length or model size. Experiments on LLaDA‑8B‑Instruct and Dream‑7B‑Instruct demonstrate speedups of 15.2×–26.4× on standard benchmarks and 22.4×–24.1× on chain‑of‑thought benchmarks while maintaining competitive accuracy and keeping decision overhead below 0.5% of inference time.
By Minsoo Cheong, Donghyun Son, Woosang Lim, Sungjoo Yoo
arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.
By Ning Ni, Yingjie Lao
AbridgeKV compresses long chain‑of‑thought outputs by encoding completed text blocks into the KV cache of specially trained pause tokens, freeing the original cache space. The method uses pause tokens positioned within the block, allocates cache budget per layer based on entry value, and compresses large blocks. Experiments on 446 competition problems show that with only a quarter of the original KV cache, AbridgeKV matches the accuracy and output length of a full‑KV model, while enabling faster inference and a 3.5× smaller cache per sequence.
By Jack Cai, Lijie Yang, Vasilis Kontonis, Dimitris Papailiopoulos, Shivam Garg
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
By Yu Li, Binxu Li, Tian Lan
arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
By Chao Fei, Panos Kalnis
arXiv:2608.21362v1 Announce Type: new
Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
By Srihari Unnikrishnan
BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.
By Jiamu Zhang, Hanjie Chen
HeadWiseKV is a training‑free framework that compresses the residual global key–value caches of hybrid long‑context language models by assigning each physical KV head a static, multilevel history window. It formulates cache allocation as a restricted operational rate–distortion problem and uses the SeqCalib algorithm to generate per‑head residency policies that account for interactions across layers. In evaluations on four hybrid models, HeadWiseKV preserves near‑full‑KV quality while reducing peak device memory usage by 8.59% at a 112K context length and extending the largest verified context from 114K to 161K.
By Renjie Xie, Juncheng Yang, Aoting Hu, Mingxi Zhang, Liyao Wu, Zheheng Hong, Wei Xu
arXiv:2602.08005v2 Announce Type: replace-cross
Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computati...
By Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li
DeferKV rethinks when to evict key‑value (KV) cache entries in long‑context large language models. By delaying eviction until the first decoding step and combining prompt‑side and decode‑side attention signals, it aligns KV importance estimation with actual generation needs. The method requires no extra training or modules and consistently improves performance on benchmarks while keeping latency low.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.