arXiv Machine Learning
Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Sep 10
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...
arXiv AI
Jun 30
KernelSight-LM: A Kernel-Level LLM Inference Simulator
arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.
arXiv AI
Jul 23
Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX
arXiv:2607. 19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets.
arXiv Machine Learning
Aug 21
RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference
arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.
arXiv Machine Learning
Jun 3
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
arXiv:2606. 02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling.
arXiv AI
Oct 15
Fewer Bits, More Tries: Recovering Capability Under Memory Constraints
arXiv:2610.15372v1 Announce Type: new Abstract: Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether p...