The paper introduces VFold, a symmetry‑aware strategy for merging value caches across layers in large language models to reduce memory usage during decoding. It avoids the performance penalties and architectural changes typical of other compression methods, and can be combined with high‑ratio quantization or key cache pruning to achieve higher compression ratios. The authors demonstrate that a significant portion of the value cache is underutilized, providing a straightforward path to scaling context windows within memory limits.
By Neha Verma, Sungwon Kim, Kenton Murray, Kevin Duh
MultiAlloc is a profiling‑guided framework that allocates limited quantization resources between layer‑wise precision enhancement and token‑wise preservation for KV cache quantization in large language models. By using lightweight offline profiling to assess layer sensitivity and token importance, it automatically derives an effective precision allocation strategy that satisfies a fixed memory budget without online search. Experiments on reasoning and long‑context benchmarks show that MultiAlloc consistently outperforms existing layer‑wise, token‑wise, and multi‑granularity quantization methods across models such as Mistral, Llama, DeepSeek‑Qwen, and QwQ.
arXiv:2608. 02901v1 Announce Type: new Abstract: The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference.
By Malik Khalaf, Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, Assaf Schuster
arXiv:2610.03027v1 Announce Type: cross
Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
By Minsoo Cheong, Donghyun Son, Sungjoo Yoo
arXiv:2609.39329v1 Announce Type: new
Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
By Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li