arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
arXiv:2606. 03910v1 Announce Type: cross Abstract: Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget.
Related stories
arXiv Machine Learning
Aug 4
HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.
arXiv AI
Jul 7
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
arXiv:2607. 02525v1 Announce Type: cross Abstract: We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime.
arXiv Machine Learning
Jun 2
Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
arXiv:2606. 01839v1 Announce Type: cross Abstract: LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives.
Hugging Face Trending Papers
Jul 5
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.
arXiv AI
Jul 1
KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving
arXiv:2605. 09735v2 Announce Type: replace-cross Abstract: Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time.
arXiv Machine Learning
Jun 3
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
arXiv AI
Jul 24
SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification
arXiv:2607. 20475v1 Announce Type: new Abstract: Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding.
arXiv AI
Aug 3
Topology-Aware Data Movement for Disaggregated GPU Inference
arXiv:2607. 28633v1 Announce Type: cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly.
arXiv Machine Learning
Jul 7
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
arXiv AI
Jun 11
Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference
arXiv:2603. 09555v2 Announce Type: replace-cross Abstract: High-throughput Mamba-2 inference is usually tied to fused CUDA and Triton kernels, limiting portability across accelerator backends.
arXiv AI
Jun 12
MiniPIC: Flexible Position-Independent Caching in <100LOC
arXiv:2606. 13126v1 Announce Type: cross Abstract: Retrieval-augmented and agentic workloads repeatedly prefill recurring predictable structured inputs (which we call "spans") such as documents and code files.