arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...
arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.
arXiv:2607. 19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets.
arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.
arXiv:2606. 02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling.
arXiv:2610.15372v1 Announce Type: new Abstract: Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether p...
PrefixBench-H100 is a reproducible benchmark that evaluates how reusing prompt prefixes affects LLM serving performance on NVIDIA H100 GPUs. It tests two popular runtimes (vLLM and TensorRT-LLM) across varied workloads, measuring metrics such as time-to-first-token, latency, throughput, cache hits, and GPU memory usage. The study identifies when prefix reuse significantly reduces first‑token latency and when cache pressure diminishes those gains, noting that cache effectiveness is largely unaffected by concurrency or output length, while differences arise mainly in scheduling.
TokenPowerSandbox is an evidence‑gated workflow that uses a CPU‑resident projector, brief GPU probes, full‑workload verification, and tamper‑evident provenance to predict energy usage of large language model serving. In experiments on an NVIDIA H100 80GB running Qwen2.5‑7B‑Instruct with vLLM, the method achieved energy MAPE of 6.23% and 7.35% on blind holdout and no‑refit confirmations, with high Spearman rank correlations. A predeclared TTFT gate demonstrated that energy accuracy alone cannot guarantee latency, as it passed at concurrency four but abstained below that level.
arXiv:2607. 20806v1 Announce Type: new Abstract: Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments.
The paper investigates how a single SM utilization metric can misrepresent the true workload of large language model (LLM) inference on Nvidia Hopper GPUs. By profiling vLLM with FlashAttention‑3 and cuBLASLt on an H100 NVL across various phases (cold prefill, warm prefill, and decode) and varying sequence length and batch size, the authors replace the single utilization figure with eight detailed counter‑validated views. These views, tied to specific Nsight Compute counters or formulas, reveal how factors such as fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection create utilization gaps across four production models and six per‑layer kernel roles.
Dooly is a profiling tool for large language model (LLM) inference that eliminates the need to re‑profile every operation for each configuration. By labeling input dimensions with taint propagation and only profiling operations missing from its latency database, Dooly builds a configuration‑agnostic, redundancy‑aware latency database that can be used as a drop‑in backend for simulators. In experiments across two GPU platforms, three attention backends, and multiple model architectures, Dooly achieves simulation accuracy within 5% MAPE for time‑to‑first‑token and 8% for total‑processing‑time while cutting profiling GPU‑hours by 56.4% compared to traditional methods.
arXiv:2607. 05475v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency.