The paper studies how evicting entries from a KV‑cache affects the token distributions used in autoregressive language models. It derives an exact decomposition showing that the expected fraction of mismatched tokens equals a first‑mismatch contribution plus a post‑divergence exposure term weighted by its mismatch rate. Experiments on Meta‑Llama‑3.1‑8B‑Instruct and Qwen2.5‑7B‑Instruct demonstrate that a 50% retention strategy (SnapKV) delays divergence and reduces its frequency compared to other retention schemes, while post‑divergence total variation remains high and dominates the overall mismatch gap.
By Xinyue Luo, Fei Yu
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
By Yu Li, Binxu Li, Tian Lan
arXiv:2610.00694v1 Announce Type: cross
Abstract: Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on t...
By Beatriz Almeida Felicio
arXiv:2610.07643v1 Announce Type: cross
Abstract: Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter whi...
By Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Wajih Hassan Raza, Atta Ul Asad, Young D. Kwon, Michal Valko, Dean F. Hougen
arXiv:2609.38096v1 Announce Type: new
Abstract: Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can requi...
By Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer, Rasul Tutunov, Haitham Bou-Ammar
arXiv:2607. 28495v1 Announce Type: new Abstract: Stage-replay diagnostics reconstruct intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.
By Alexander Boesgaard Lorup
arXiv:2609.25735v1 Announce Type: new
Abstract: Class-balanced replay controls class frequency but does not determine the interval between successive replay appearances of a class. We study this inte...
By Congren Dai, Nat Roongjirarat, Fei Ye
arXiv:2609.07901v1 Announce Type: new
Abstract: Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4...
By Dachi Kurtskhalia
arXiv:2607. 21475v1 Announce Type: cross Abstract: Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.
By Peng Xie
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
By Junyoung Park, Jungwook Choi, Mingu Lee
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
By Stephen Gould, Anton van den Hengel
arXiv:2608. 13756v1 Announce Type: new Abstract: Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.
By Teng-Ruei Chen