GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Read the original on arXiv AI →GAUSS is a workload‑aware analytical and Monte‑Carlo simulator designed to predict the full distributions of time‑to‑first‑token (TTFT) and time‑between‑tokens (TBT) for large language model (LLM) inference serving. It combines calibrated, phase‑specific latency surfaces for prefill and decode with a Lindley‑recurrence queueing model, using ensemble Monte‑Carlo sampling to estimate performance rather than replaying a single trace. This approach enables operators to evaluate workloads, routing policies, and hardware configurations that have never been deployed, and to capture latency tails that trace‑driven simulators typically miss, thereby supporting operating‑point selection, traffic segmentation, and scheduling capacity decisions.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.