arXiv:2606. 00516v1 Announce Type: new Abstract: Mixed batching (MB)--interleaving prefill and decode in a single batch--has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization.
arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes t...
arXiv:2504. 11320v4 Announce Type: replace-cross Abstract: Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day.
arXiv:2609.13592v1 Announce Type: cross Abstract: GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-...
arXiv:2606. 17104v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge.
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.
arXiv:2607. 08930v1 Announce Type: new Abstract: Efficient serving of diffusion large language models (dLLMs) is hindered by convergence heterogeneity: when batching multiple requests, different sequences converge at different rates, causing faster requests to stall behind slower stragglers and introducing compute bubbles and tail latency.
Masked diffusion language models (dLLMs) promise faster text generation by denoising multiple tokens simultaneously, yet their real‑world serving behavior has been largely unexamined. Using LLaDA‑8B‑Instruct on a single NVIDIA H200 GPU, the study finds that request difficulty is discretized into 11 step‑count levels, short‑budget benchmarks underestimate serving variance, and only 24% of single‑request time is GPU computation, with batching mainly reducing CPU dispatch overhead. The authors also demonstrate that output quality remains stable across batch sizes and propose a batch‑timeout rule for synchronized batching under Poisson arrivals.
arXiv:2608. 09444v1 Announce Type: new Abstract: A main promise of looped language models (LMs) is depth-adaptive inference.
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.