arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv:2607. 24331v1 Announce Type: new Abstract: As the inference phase of Large Language Models (LLMs) requires handling long context windows, the Key-Value (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context window continues to grow.
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.
arXiv:2606. 21848v2 Announce Type: replace-cross Abstract: Transformer architectures form the foundation of modern natural language processing, making it crucial to address the efficiency and scalability limitations of the standard QKV attention mechanism.
The paper introduces CORE, a method that jointly calibrates coverage and redistributes evicted mass for key–value (KV) cache in long‑context decoding. CORE combines query utility and log‑determinant coverage into a lightweight indexer, using a single calibrated distribution to both retain complementary KV states and parameterize latent‑memory writes. Experiments on three backbones show CORE outperforms the RULER baseline by up to 3.78 points at 90% compression and demonstrates strong decoding efficiency on LongBench and repeated‑eviction tests.
arXiv:2610.17966v1 Announce Type: cross Abstract: Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compr...
The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
arXiv:2608. 14191v1 Announce Type: new Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference.
arXiv:2610.28385v1 Announce Type: new Abstract: Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional represen...
Dual-QK introduces paired non‑orthogonal transforms for queries and keys to enable sharp queries and flat keys, improving 2‑bit quantization and dynamic channel pruning of KV caches. The method balances key scales for INT2 quantization while concentrating query energy for pruning, and supports channel‑0 protection and bucket‑relative RoPE for long‑context accuracy. Experiments on four models across five generative benchmarks show that Dual‑QK outperforms OSCAR at 40% query‑channel sparsity, achieving 6.8× KV‑cache compression, an 8.3× reduction in KV read volume, and up to 3.75× decoding throughput over unpruned BF16.