Hugging Face Trending Papers

Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

Read the original on Hugging Face Trending Papers →

The paper investigates how prefix caching in LLM agent serving can lead to persistent degradation when traffic bursts occur. It shows that even a small increase in arrivals can push a vLLM server into a low hit‑ratio, low‑throughput regime that continues to deteriorate for an extended period, while a server without prefix caching remains stable. The authors identify a workload‑balance condition that predicts this collapse and evaluate mitigation strategies such as throttling, session suspension, and a KV‑budgeting approach, finding the latter most effective.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI
6d ago

Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

The paper investigates how prefix caching in large language model (LLM) agents can lead to persistent degradation when small bursts of additional requests occur. Even with identical base workloads, a brief surge can push a vLLM server into a degraded state—reducing hit ratios, throughput, and causing a backlog that grows for over an hour—while a server without the burst remains healthy. The study identifies that slower service keeps more sessions active, expanding the working set beyond the key‑value pool and triggering repeated recomputations, and demonstrates that simple mitigation strategies such as admission throttling, session suspension, and a budgeter that limits resident KV demand can prevent or recover from this collapse.

By Wen Huang, Jack Liu
arXiv Machine Learning
Sep 7

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

The paper investigates how prefix caching, a default optimization in open‑source LLM serving stacks, affects reproducibility when combined with weight quantization. Experiments on an eighty‑episode multi‑turn agentic tool‑use workload show that enabling the cache causes the agent’s trajectory to change in 36.2 % of episodes at 16‑bit precision and 75.0 % at 4‑bit precision, while disabling the cache yields perfectly reproducible runs. The study identifies specific cache‑related settings that drive run‑to‑run divergence and demonstrates that cached serving is deterministic only when the cache state is preserved, which is not the case in typical deployments.

By Aditi Patodiya
arXiv Machine Learning
Oct 16

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %. whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."

By Muhammad Ibrahim
arXiv AI
Aug 26

Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap

The paper introduces an elastic key‑value (KV) cache for large language model (LLM) serving that dynamically reclaims a pre‑allocated reserve during decode‑heavy phases and restores it before prefill, using a userspace CUDA virtual‑memory trick that requires no driver changes. The authors implement this mechanism, test it under realistic workloads, and find that it offers only marginal benefits—about a 1 % difference in time‑to‑first‑token for large prefill chunks—and that simpler strategies such as lowering the maximum batch size can achieve similar results. The study also notes that the reserve’s impact diminishes with higher tensor‑parallelism levels. whyItMatters":"The work demonstrates that a dynamic KV cache reclamation strategy can be implemented without driver patches and that its practical benefits are limited, guiding future LLM serving optimizations toward simpler approaches."

By Sathishkumar Sivashanmugam
arXiv Machine Learning
Nov 5

A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

The study evaluates how configuration choices, particularly chunked prefill, affect performance differences between two large language model serving engines, vLLM and SGLang. Using a reproducible Kubernetes testbed and a seeded multi‑agent workload, the authors show that matching the chunked prefill setting eliminates a large portion of the observed capacity and latency gaps. Even after matching, vLLM still exhibits a tail‑latency advantage due to cache‑aware request ordering, and the authors provide six recommendations for fair engine comparisons.

By Muhammad Faizan Anwar, Sina Keshvadi