The paper investigates the space complexity of attention mechanisms when key/query cosine similarities are constrained to a small interval around zero, reflecting the anisotropy observed in real transformers. It presents a nearly space‑optimal algorithm alongside a matching lower bound, showing that the distribution of similarities in the lower bound aligns with empirical anisotropy. The study also demonstrates that a simple uniform sampling approach is nearly optimal for typical parameter ranges, explaining its effectiveness on real datasets.
arXiv:2610.28385v1 Announce Type: new Abstract: Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional represen...
arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.
arXiv:2610.10135v1 Announce Type: cross Abstract: Sparse attention mechanisms estimate attention over $n$ tokens using a small subset of keys. Many existing approaches use maximum inner product searc...
The paper introduces QCOC, a method for compressing the key‑value cache used in tabular in‑context learning. QCOC clusters in‑context example states into joint‑KV prototypes, calibrates them against query vectors, and preserves multiplicities to maintain accuracy while achieving high compression. Experiments on 64 OpenML‑CC18 datasets and seven long tables show QCOC outperforms other compression and retrieval methods, achieving up to a 10.5× compression ratio and 508× speed‑up over dynamic retrieval baselines while staying within 0.23 percentage points of full‑context accuracy.
The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
arXiv:2609.13285v1 Announce Type: cross Abstract: The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query a...
arXiv:2608. 03276v1 Announce Type: new Abstract: Long-context inference with large language models is constrained by the linear growth of the key-value cache to sequence length.
arXiv:2607. 24331v1 Announce Type: new Abstract: As the inference phase of Large Language Models (LLMs) requires handling long context windows, the Key-Value (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context window continues to grow.
arXiv:2609.36835v1 Announce Type: new Abstract: Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for...
arXiv:2607. 27692v1 Announce Type: cross Abstract: Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries.
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
Dual-QK introduces paired non‑orthogonal transforms for queries and keys to enable sharp queries and flat keys, improving 2‑bit quantization and dynamic channel pruning of KV caches. The method balances key scales for INT2 quantization while concentrating query energy for pruning, and supports channel‑0 protection and bucket‑relative RoPE for long‑context accuracy. Experiments on four models across five generative benchmarks show that Dual‑QK outperforms OSCAR at 40% query‑channel sparsity, achieving 6.8× KV‑cache compression, an 8.3× reduction in KV read volume, and up to 3.75× decoding throughput over unpruned BF16.