Divergence Timing and Cumulative Disagreement under KV-Cache Eviction
Read the original on arXiv Machine Learning →The paper studies how evicting entries from a KV‑cache affects the token distributions used in autoregressive language models. It derives an exact decomposition showing that the expected fraction of mismatched tokens equals a first‑mismatch contribution plus a post‑divergence exposure term weighted by its mismatch rate. Experiments on Meta‑Llama‑3.1‑8B‑Instruct and Qwen2.5‑7B‑Instruct demonstrate that a 50% retention strategy (SnapKV) delays divergence and reduces its frequency compared to other retention schemes, while post‑divergence total variation remains high and dominates the overall mismatch gap.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.