arXiv Machine Learning
Oct 5

Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention

Page-EntroKV is a new KV‑cache eviction framework designed for grouped‑query attention (GQA) in long‑context autoregressive language models. It pools token importance scores across query heads within each physical group using sink‑isolated Renyi‑2 entropy, then projects these pooled scores onto page frames and performs eviction at the hardware tuple level. The authors prove theoretical guarantees on union overhead, budget preservation, and needle‑retention, and demonstrate on a Qwen2.5‑1.5B‑Instruct pilot that Page‑EntroKV achieves exact union overhead control, eliminates sink masquerade, and retains 100% of critical tokens compared to mean‑pooling baselines.

By Inbasekaran S
arXiv Machine Learning
Oct 16

HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

HAServe introduces a horizon-aware scheduling and memory co-design framework for efficient large language model (LLM) serving. By modeling request admission and key-value (KV) retention as a unified problem using the HorizonSchema abstraction, it selects joint admission and KV-retention candidates that minimize predicted latency across varying prefix structures and arrival rates. Experiments on Qwen3-4B-Instruct-2507 and Llama2-13B with eight H100 GPUs demonstrate up to 5.90× P99 TTFT speedups, 1.44× throughput gains, and a 20.59 percentage point increase in KV-cache hit rate over the strongest baseline.

By Xingyu Qu, Tianhao Lin, Yiqi Li, Zhiyu Chen, Sheng Wang
arXiv Computation and Language
Nov 4

Pacing Branch Parallelism in LLM Serving

The paper introduces PACE, a per‑step admission controller that manages intra‑request parallelism in large language model serving. By admitting extra decoding branches only when the predicted branch externality fits within the batch’s current slack, PACE avoids the pitfalls of eager or fixed‑cap branch admission. Experiments on Qwen3‑32B and other large models show that PACE boosts goodput by up to 2.15× while maintaining 99% SLO attainment.

By Swapnil Gandhi, Siva Kumar Sastry Hari, William J. Dally, Christos Kozyrakis