arXiv AI
Aug 3
Topology-Aware Data Movement for Disaggregated GPU Inference
arXiv:2607. 28633v1 Announce Type: cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly.
The Price of Anarchy in Disaggregated Inference
arXiv:2606. 17081v1 Announce Type: cross Abstract: Disaggregated inference architectures physically separate prefill and decode phases onto distinct GPU pools, creating competing "agents" that share a fixed hardware budget.
Related stories
arXiv AI
6d ago
Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
arXiv:2608. 12123v1 Announce Type: cross Abstract: LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect.
arXiv AI
Jun 2
Lodestar: An Online-Learning LLM Inference Router
arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.
arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
arXiv AI
Jun 2
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
arXiv:2606. 01502v1 Announce Type: cross Abstract: Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk.
arXiv AI
Jun 16
Service-Induced Congestion in Memory-Constrained LLM Serving
arXiv:2606. 15555v1 Announce Type: cross Abstract: In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token.
arXiv AI
Jul 8
Think Before You Grid-Search: Floor-First Triage for LLM Serving
arXiv:2607. 05876v1 Announce Type: cross Abstract: LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed.
arXiv AI
Jun 24
CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
arXiv:2606. 24506v1 Announce Type: cross Abstract: Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold.
arXiv AI
Jul 21
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
arXiv:2607. 17181v1 Announce Type: cross Abstract: Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently.
arXiv Machine Learning
Jul 14
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
arXiv:2607. 11368v1 Announce Type: cross Abstract: Reported serving speedups from quantized kernels typically bundle the weight format, the kernel, and the inference runtime into one number.
arXiv Machine Learning
Aug 4
HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.
arXiv Machine Learning
Jun 2
Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
arXiv:2606. 01839v1 Announce Type: cross Abstract: LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives.