arXiv AI September 18, 2026 By Chao Fei, Kaihua Liang, Hanzhi Hu, Hongcheng Guo, Jian Weng, Marco Canini, Panos Kalnis

Exploring a Layer-Wise Design Space for KV Cache Eviction

The paper investigates whether key‑value (KV) cache eviction strategies should vary across Transformer layers. By combining existing eviction methods in different layer configurations and profiling their performance, the authors find that heterogeneous, layer‑wise routing consistently outperforms homogeneous policies on LongBench tasks. Even with a fixed set of methods, the placement of each method strongly influences overall quality, and a single well‑chosen route surpasses all nine standalone baselines across multiple cache budgets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 16

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.

By Chao Fei, Panos Kalnis

llms efficiency

arXiv AI

Aug 11

RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.

By Dongjie Xu, Kai Qian, Julius, Weijie Shi, Yuxuan Sun, Minghua Tang, Fenglei Jin, Hanchi Dong, Jiajie Xu

llms efficiency

arXiv Machine Learning

Jun 15

Sub-Token Routing for KV Cache Compression

arXiv:2604. 21335v3 Announce Type: replace Abstract: Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation.

By Wei Jiang, Wei Wang

llms efficiency multimodal

arXiv Machine Learning

Aug 10

Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

arXiv:2608. 07001v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck.

By Haolin Tian, Yuzhe Liu, Tonghan Wang

llms efficiency

arXiv AI

4d ago

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...

By Joao Monteiro, Louis B\'ethune, Anastasiia Filippova, Sonia Laguna, David Grangier, Marco Cuturi

llms efficiency

arXiv Machine Learning

Jul 3

The risk of KV cache compression

arXiv:2607. 01520v1 Announce Type: new Abstract: Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache.

By Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno, Sebastian Trimpe, Marco Pavone

llms efficiency

One email a morning, machine-written

Related stories

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation

Sub-Token Routing for KV Cache Compression

Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

The risk of KV cache compression