arXiv Machine Learning August 10, 2026 By Haolin Tian, Yuzhe Liu, Tonghan Wang

Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

Read the original on arXiv Machine Learning →

arXiv:2608. 07001v1 Announce Type: new Abstract: As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

4d ago

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...

By Joao Monteiro, Louis B\'ethune, Anastasiia Filippova, Sonia Laguna, David Grangier, Marco Cuturi

llms efficiency

arXiv AI

Jun 16

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.

By Chao Fei, Panos Kalnis

llms efficiency

arXiv Machine Learning

Jul 2

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

arXiv:2607. 00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost.

By Sheng Qiang, Ruiwei Chen, Yinpeng Wu, Jinyu Gu, Zhichao Hua, Yubin Xia, Binyu Zang, Haibo Chen

llms efficiency

arXiv Machine Learning

Aug 26

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...

By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li

llms efficiency benchmarks

arXiv AI

Jun 6

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.

By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu

llms fine-tuning efficiency

arXiv Machine Learning

Sep 17

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.

By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu

llms diffusion efficiency

One email a morning, machine-written

Related stories

KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference