arXiv Machine Learning June 9, 2026 By Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikci, Yang Zhou, Jiarong Xing, Ion Stoica

BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

Read the original on arXiv Machine Learning →

arXiv:2411. 16102v2 Announce Type: replace Abstract: Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

1d ago

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.

By Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani

llms efficiency

arXiv AI

Jul 13

Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

arXiv:2607. 09172v1 Announce Type: cross Abstract: Large Language Models are reshaping how software is developed and maintained.

By Nada Zine, Tristan Coignion, Vincenzo Stoico, Cl\'ement Quinton, Romain Rouvoy, Patricia Lago

llms

arXiv AI

Jun 9

Harmonia: End-to-End RAG Serving Optimization

arXiv:2505. 07833v2 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) improves the reliability of large language models by integrating external knowledge, but serving RAG pipelines efficiently is challenging because requests traverse heterogeneous components spanning LLM inference, databases, and CPU-side processing.

By Saurabh Agarwal, Bodun Hu, Luis Pabon, Myungjin Lee, Jayanth Srinivasa, Aditya Akella

llms rag efficiency

arXiv AI

Jun 30

LLM Serving Optimization with Variable Prefill and Decode Lengths

arXiv:2508. 06133v4 Announce Type: replace-cross Abstract: We study offline scheduling for large language model (LLM) serving under a fixed KV-cache memory budget, where requests have heterogeneous prompt (prefill) and response (decode) lengths.

By Meixuan Wang, Yinyu Ye, Zijie Zhou

llms nlp

arXiv AI

Jul 22

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

arXiv:2607. 18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost.

By Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick, Gauri Joshi

llms

arXiv Machine Learning

Aug 3

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

arXiv:2607. 28848v1 Announce Type: cross Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak.

By Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, Oana Balmau

llms fine-tuning efficiency

Related stories

Multi-Bin Batching for Increasing LLM Inference Throughput

Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

Harmonia: End-to-End RAG Serving Optimization

LLM Serving Optimization with Variable Prefill and Decode Lengths

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs