arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
The paper investigates whether key‑value (KV) cache eviction strategies should vary across Transformer layers. By combining existing eviction methods in different layer configurations and profiling their performance, the authors find that heterogeneous, layer‑wise routing consistently outperforms homogeneous policies on LongBench tasks. Even with a fixed set of methods, the placement of each method strongly influences overall quality, and a single well‑chosen route surpasses all nine standalone baselines across multiple cache budgets.
arXiv:2608. 07001v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck.
arXiv:2607. 01520v1 Announce Type: new Abstract: Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache.
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
arXiv:2604. 21335v3 Announce Type: replace Abstract: Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation.