arXiv Machine Learning August 18, 2026 By Anders Vestrum, Arya Raeesi, Hanna Roed

Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving

Read the original on arXiv Machine Learning →

arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse

llms agents safety

arXiv AI

Jul 28

HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

arXiv:2607. 22578v1 Announce Type: new Abstract: The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows.

By Size Li, Zhiqing Tang, Hongrui Liang, Jianxiong Guo, Jiong Lou, Tian Wang, Weijia Jia

llms agents efficiency

arXiv AI

Jun 16

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.

By Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang, Yibo Zhu, Hong Xu

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Aug 3

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

arXiv:2607. 28848v1 Announce Type: cross Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak.

By Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, Oana Balmau

llms fine-tuning efficiency

arXiv AI

Jun 29

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

arXiv:2510. 03243v3 Announce Type: replace-cross Abstract: Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable.

By Yiheng Tao, Yihe Zhang, Matthew Dearing, Xin Wang, Yuping Fan, Michael E. Papka, Zhiling Lan

llms efficiency benchmarks

arXiv AI

Jul 23

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.

By Tiancheng Zhang, Shaoyuan Huang, Mingyuan Wang, Yunfeng Zhao, Xiaofei Wang, Wenyu Wang

llms efficiency benchmarks

Related stories

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads