The paper studies how evicting entries from a KV‑cache affects the token distributions used in autoregressive language models. It derives an exact decomposition showing that the expected fraction of mismatched tokens equals a first‑mismatch contribution plus a post‑divergence exposure term weighted by its mismatch rate. Experiments on Meta‑Llama‑3.1‑8B‑Instruct and Qwen2.5‑7B‑Instruct demonstrate that a 50% retention strategy (SnapKV) delays divergence and reduces its frequency compared to other retention schemes, while post‑divergence total variation remains high and dominates the overall mismatch gap.
KV-cache eviction perturbs the conditional token distributions governing autoregressive generation. We investigate how first-divergence timing and subsequent token mismatch determine cumulative disagr...
arXiv:2610.00694v1 Announce Type: cross Abstract: Compression reports summarize how far a compressed language model moved from the dense one, usually by a KL divergence; a deployment that relies on t...
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
arXiv:2610.07643v1 Announce Type: cross Abstract: Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter whi...
arXiv:2609.38096v1 Announce Type: new Abstract: Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can requi...
arXiv:2607. 28495v1 Announce Type: new Abstract: Stage-replay diagnostics reconstruct intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.
arXiv:2607. 16811v4 Announce Type: replace Abstract: Drift detectors that work tend not to explain themselves, and drift detectors that explain themselves tend to fail in high dimension.
arXiv:2607. 20129v1 Announce Type: new Abstract: Quantized small autoregressive reasoning models can enter long, repetitive, or unproductive trajectories, yet inference-time compute is usually allocated without observing how a trajectory develops.
arXiv:2608. 13756v1 Announce Type: new Abstract: Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.
arXiv:2609.07901v1 Announce Type: new Abstract: Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4...
arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.
arXiv:2607. 21475v1 Announce Type: cross Abstract: Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.