arXiv AI
From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 11
Optimizing AI Inference Across the Deployment Stack
The paper argues that AI deployment performance depends on interactions among compression, compiler transformations, and serving policies rather than just model architecture. It introduces a three‑layer taxonomy—model‑level techniques, compiler transformations, and system policies—and frames deployment as a constrained multi‑objective optimization problem over accuracy, latency, throughput, memory footprint, and energy. The authors propose an evidence protocol for comparable benchmarking and synthesize data from edge and data‑center platforms to show that cross‑layer interactions drive deployment outcomes, concluding with a constraint‑aware selection procedure and open research problems.
arXiv AI
Jun 16
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.
arXiv Machine Learning
Aug 18
Multi-Bin Batching for Increasing LLM Inference Throughput
arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.
arXiv Machine Learning
Aug 18
Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.
arXiv Machine Learning
Aug 3
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
arXiv:2607. 28848v1 Announce Type: cross Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak.
arXiv AI
Jul 24
InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents
arXiv:2607. 20468v1 Announce Type: new Abstract: AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces.