arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
FluidPD is a prefill‑decode disaggregated LLM serving system that introduces in‑place elasticity to meet SLOs. It uses FluidToken to offload a bounded portion of prefill work to decode workers during short‑term imbalance, and FluidRole to reassign running workers between prefill and decode roles for sustained imbalance, all guided by lightweight pressure indices. In production Azure trace workloads, FluidPD improves overall SLO attainment over static SGLang by up to 94.6 percentage points without adding workers.
Related stories
arXiv AI
Sep 24
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Crossflow introduces an elastic boundary for prefilling and decoding in large language model serving, allowing decode nodes to publish short‑lived leases that limit prefilling resources and output projections. By adapting to dynamic phase demand, Crossflow improves token throughput by 16.2‑17.4% on average and up to 43.4% under high load, while consistently reducing mean time‑to‑first‑token. The approach eliminates the inefficiencies of static partitioning, which can leave 17% of cluster capacity idle or cause queueing and lost throughput.
arXiv Machine Learning
Aug 10
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.
arXiv AI
Jul 23
FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.
arXiv Machine Learning
Aug 18
Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.
arXiv Computation and Language
Aug 27
TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
TOPAS is a Task‑Oriented Prefix‑Aware Scheduler designed for multi‑agent large language model serving. It jointly decides which agent prefixes to retain in a shared key‑value cache and which requests to schedule, balancing the reduction of each task’s longest remaining service path against the benefit of downstream prefix reuse while accounting for movement and preemption costs. Experiments on synthetic DAGs and MetaGPT software‑development workflows show that TOPAS can reduce mean and p99 job completion times by up to 39.8%/49.4% and 22.0%/26.6% respectively compared to the best baselines.
arXiv Machine Learning
Jul 7
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
arXiv AI
Jul 10
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
arXiv:2607. 08565v1 Announce Type: cross Abstract: LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans.
Hugging Face Trending Papers
Jul 5
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.
arXiv Machine Learning
Oct 16
HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving
HAServe introduces a horizon-aware scheduling and memory co-design framework for efficient large language model (LLM) serving. By modeling request admission and key-value (KV) retention as a unified problem using the HorizonSchema abstraction, it selects joint admission and KV-retention candidates that minimize predicted latency across varying prefix structures and arrival rates. Experiments on Qwen3-4B-Instruct-2507 and Llama2-13B with eight H100 GPUs demonstrate up to 5.90× P99 TTFT speedups, 1.44× throughput gains, and a 20.59 percentage point increase in KV-cache hit rate over the strongest baseline.
arXiv AI
2d ago
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
FlashOffloading is a host‑DRAM‑centric inference architecture that fully offloads the key‑value (KV) cache for large language model serving, addressing the high‑bandwidth memory (HBM) capacity wall that limits concurrency. It introduces a rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero miss‑lookup overhead, and a compute–load overlapped kernel to reduce KV offloading latency across the pipeline. Experiments on an Ascend910C platform with GLM‑5.2 show up to 1.38× prefill throughput and 1.91× decode throughput gains under a 30 ms time‑per‑output‑token constraint.
arXiv Machine Learning
3d ago
A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines
arXiv:2611.05991v1 Announce Type: cross Abstract: Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM...