arXiv Machine Learning July 27, 2026 By Chao Fang, Jun Yin, Man Shi, Marian Verhelst

HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

Read the original on arXiv Machine Learning →

arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 2

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

arXiv:2607. 00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost.

By Sheng Qiang, Ruiwei Chen, Yinpeng Wu, Jinyu Gu, Zhichao Hua, Yubin Xia, Binyu Zang, Haibo Chen

llms efficiency

arXiv Machine Learning

Jun 3

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.

By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui

llms rag agents efficiency

arXiv AI

Aug 25

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...

By Srihari Unnikrishnan

llms efficiency

arXiv Computation and Language

Sep 23

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

Flash-dLLM is a training‑free inference acceleration framework that improves the speed and memory efficiency of Diffusion Large Language Models (dLLMs). It tackles GPU memory I/O bottlenecks by introducing an I/O‑aware fused KV‑cache kernel and then employs a draft‑and‑verify decoding strategy that uses the dLLM itself as both drafter and verifier. Experiments on mathematical reasoning and code‑generation tasks show Flash‑dLLM outperforms existing acceleration methods, achieving up to 11.0× speedups over the Elastic‑Cache baseline.

By Quan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen

llms diffusion efficiency benchmarks

arXiv Machine Learning

Jun 5

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.

By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi

llms efficiency

arXiv AI

Jun 30

HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

arXiv:2606. 28831v1 Announce Type: cross Abstract: Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.

By Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li

llms efficiency benchmarks

One email a morning, machine-written

Related stories

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving

HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression