arXiv AI
Jun 16
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
GAUSS is a workload‑aware analytical and Monte‑Carlo simulator designed to predict the full distributions of time‑to‑first‑token (TTFT) and time‑between‑tokens (TBT) for large language model (LLM) inference serving. It combines calibrated, phase‑specific latency surfaces for prefill and decode with a Lindley‑recurrence queueing model, using ensemble Monte‑Carlo sampling to estimate performance rather than replaying a single trace. This approach enables operators to evaluate workloads, routing policies, and hardware configurations that have never been deployed, and to capture latency tails that trace‑driven simulators typically miss, thereby supporting operating‑point selection, traffic segmentation, and scheduling capacity decisions.
Related stories
arXiv AI
Oct 16
Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.
arXiv Machine Learning
Oct 7
Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads
arXiv:2610.07094v1 Announce Type: cross Abstract: LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test...
arXiv AI
Jul 23
FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads
arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.
arXiv AI
Jun 30
KernelSight-LM: A Kernel-Level LLM Inference Simulator
arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.
arXiv AI
Jun 16
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
arXiv:2504. 11320v4 Announce Type: replace-cross Abstract: Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day.
arXiv Machine Learning
Oct 15
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
The paper investigates how different parallelism strategies—tensor parallelism (TP), pipeline parallelism (PP), and hybrids—affect the trade‑off between time‑to‑first‑token (TTFT) and inter‑token latency (ITL) in large language model inference. It introduces a framework that maps feasible TTFT‑ITL regions and Pareto frontiers for various workloads, revealing a shift from TP‑dominant to PP‑dominant configurations as prefill work increases. The authors also present an SLO‑aware selection method based on goodput, showing that stricter TTFT goals favor PP, while relaxed constraints move toward TP, and they demonstrate that longer contexts further favor PP for scalability and SLO attainment.
arXiv Machine Learning
Aug 21
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
arXiv:2608. 19659v1 Announce Type: new Abstract: Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load.
arXiv AI
Jun 2
Lodestar: An Online-Learning LLM Inference Router
arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.
arXiv AI
Oct 15
Stable Token-level Scheduling for Serverless LLM Inference
arXiv:2610.17689v1 Announce Type: new Abstract: Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, i...
arXiv AI
Oct 23
Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
Dooly is a profiling tool for large language model (LLM) inference that eliminates the need to re‑profile every operation for each configuration. By labeling input dimensions with taint propagation and only profiling operations missing from its latency database, Dooly builds a configuration‑agnostic, redundancy‑aware latency database that can be used as a drop‑in backend for simulators. In experiments across two GPU platforms, three attention backends, and multiple model architectures, Dooly achieves simulation accuracy within 5% MAPE for time‑to‑first‑token and 8% for total‑processing‑time while cutting profiling GPU‑hours by 56.4% compared to traditional methods.
arXiv AI
1d ago
Uncertainty-Aware Resource Orchestration for Agentic AI Systems
The paper introduces Conformal Reservation for Orchestration (CRO), an online algorithm that uses past execution traces to allocate compute and bandwidth for agentic AI systems at dispatch time. CRO provides finite‑sample, distribution‑free coverage guarantees, ensuring that resource reservations meet specified probabilistic deadlines. Simulations demonstrate that CRO uniquely satisfies target coverage across all risk levels while reducing over‑reservation—and thus resource waste—by about half compared to a Chebyshev‑bound baseline.