The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2607. 12550v1 Announce Type: new Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference.
The paper introduces JoLT, a training‑free compressor that jointly allocates rank and precision for key‑value (KV) cache compression in long‑context language models. JoLT treats grouped prefill caches as fourth‑order tensors, applies partial Tucker decomposition along token and feature modes, and uses a rotated low‑bit quantizer for residuals, all governed by a single Lagrangian dual under a global byte constraint. Across five models from four architecture families, JoLT achieves 2–3× compression with less than 0.2% perplexity loss, and near‑lossless retrieval accuracy on LLaMA‑3.1‑8B at 64K context up to 3× compression.
arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...
arXiv:2608. 07915v1 Announce Type: new Abstract: Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns.
Kronecker Product Attention (KPA) is introduced as a memory‑efficient method for compressing the key‑value cache in autoregressive models. By representing key and value activations as sums of Kronecker products of compact, token‑dependent matrix factors, KPA generalizes Tensor Product Attention and can capture richer block‑structured patterns. Empirical results show that KPA matches or surpasses existing techniques such as MHA, MQA, GQA, and MLA across multiple model scales and benchmarks, achieving up to 10× compression while preserving relative‑position information through a structured rotary embedding.
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...