arXiv:2608. 15584v1 Announce Type: cross Abstract: Production paged-serving engines apply uniform paging granularity to the KV cache, even though the two regions of a multi-agent workload have opposite storage requirements: a long shared prefix demands contiguity, while the per-request suffix demands fine-grained allocation.
Production paged-serving engines apply uniform paging granularity to the KV cache, even though the two regions of a multi-agent workload have opposite storage requirements: a long shared prefix demand...
arXiv:2606. 01502v1 Announce Type: cross Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk.
The paper investigates how prefix caching, a default optimization in open‑source LLM serving stacks, affects reproducibility when combined with weight quantization. Experiments on an eighty‑episode multi‑turn agentic tool‑use workload show that enabling the cache causes the agent’s trajectory to change in 36.2 % of episodes at 16‑bit precision and 75.0 % at 4‑bit precision, while disabling the cache yields perfectly reproducible runs. The study identifies specific cache‑related settings that drive run‑to‑run divergence and demonstrates that cached serving is deterministic only when the cache state is preserved, which is not the case in typical deployments.
arXiv:2609.37626v1 Announce Type: cross Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
Page-EntroKV is a new KV‑cache eviction framework designed for grouped‑query attention (GQA) in long‑context autoregressive language models. It pools token importance scores across query heads within each physical group using sink‑isolated Renyi‑2 entropy, then projects these pooled scores onto page frames and performs eviction at the hardware tuple level. The authors prove theoretical guarantees on union overhead, budget preservation, and needle‑retention, and demonstrate on a Qwen2.5‑1.5B‑Instruct pilot that Page‑EntroKV achieves exact union overhead control, eliminates sink masquerade, and retains 100% of critical tokens compared to mean‑pooling baselines.
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
arXiv:2608. 11231v1 Announce Type: new Abstract: LLM serving is increasingly accelerated by position-independent caching (PIC).
arXiv:2606. 13361v1 Announce Type: new Abstract: Right now, across the world, AI agents are repeating the same absurd act: to read one document, they each recompute it from scratch.
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
The paper introduces GraphTok, a GPU-based tokenizer that executes the entire tokenization pipeline as a single CUDA graph, matching reference token IDs for models such as GPT‑2, Llama 3, Qwen 2.5, DeepSeek‑V3, and Gemma 3. GraphTok achieves significant speedups, encoding one million GPT‑2 tokens in 1.7–3.2 ms and outperforming the leading CPU tokenizer gigatoken by 4.7–6.0× on Llama 3. In a vLLM deployment with cached prefixes, it reduces mean time‑to‑first‑token by 2.1–2.4× and dramatically increases request throughput, sustaining 518 req/s of 121k‑token traffic versus 296 with gigatoken.
arXiv:2606. 20474v1 Announce Type: cross Abstract: Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized.
arXiv:2607. 05876v1 Announce Type: cross Abstract: LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed.