arXiv AI September 4, 2026 By Qiankun Ma, Yanjiang Zhou, Zinan Xiong, Haofei Wang, Zhen Song, Yang Xiang, Ziyao Zhang, Hairong Zheng

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

Read the original on arXiv AI →

GrowPage is an on‑demand key–value (KV) budgeting framework designed to improve large language model (LLM) reasoning serving. It treats KV capacity as a runtime resource, using lightweight dual‑timescale query summaries to track recent and long‑term attention patterns and estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page, integrating with PagedAttention’s page‑level memory abstraction to maintain continuous batching and prefix caching.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 3

Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

arXiv:2607. 01237v1 Announce Type: cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.

By Shen Han, Yuyang Wu

llms efficiency

arXiv AI

Jul 7

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

arXiv:2607. 01237v2 Announce Type: replace-cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.

By Shen Han, Yuyang Wu, Junpu Yu, Olexandr Isayev

llms efficiency

arXiv AI

1d ago

Adaptive Mass-Segmented KV Compression for Long-Context Reasoning

arXiv:2605.23200v2 Announce Type: replace-cross Abstract: The linear growth of the Key-Value (KV) cache is a critical bottleneck in long-form LLM inference. Existing KV compression methods mitigate t...

By Junzhe Yang, Xiaoyu Shen

llms efficiency

Hugging Face Trending Papers

Jun 25

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce KV-cache fragmentation and mature kernels such as FlashInfer provide highly optimized native-paged decode attention.

llms efficiency

Hugging Face Trending Papers

Jun 9

ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models

Long chain-of-thought (CoT) trajectories in large language model (LLM) reasoning cause severe inference bottlenecks due to rapid key-value (KV) cache growth. Current decoding-time compression methods mitigate this issue via token eviction, but typically assume a uniform budget distribution across all layers and heads.

llms efficiency benchmarks

arXiv Machine Learning

Sep 7

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.

By Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi

rag efficiency benchmarks

One email a morning, machine-written

Related stories

Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

Adaptive Mass-Segmented KV Compression for Long-Context Reasoning

PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference