Hugging Face Trending Papers
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Sep 17
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
The paper introduces FairInference, a system that guarantees token-level latency isolation for well-behaved clients in multi-tenant LLM serving. It provides a δ-token fairness guarantee, ensuring that a token generated in isolation within time d will be produced within d + δ in a shared environment. The approach enforces per-token deadlines, bounds GPU compute sharing delays, and accounts for shared KV cache overhead, leading to reduced latency spikes and higher overall throughput compared to existing LLM serving systems.
arXiv AI
Sep 24
EMA: Elastic and Performance Transparent Memory Across GPUs
arXiv:2609.27040v1 Announce Type: cross Abstract: Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects....
arXiv Machine Learning
Jun 3
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
arXiv:2606. 02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling.
arXiv AI
Jul 7
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.
arXiv AI
Jul 10
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
arXiv:2607. 08565v1 Announce Type: cross Abstract: LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans.
arXiv AI
Jul 31
ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform
arXiv:2607. 26566v1 Announce Type: cross Abstract: Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently.