arXiv Computation and Language

TokenRouter: Efficient Serving System for Token-Level LLM Routing

TokenRouter is a new serving system designed for token‑level routing of large language models (LLMs). It adopts a request‑centric programming model and launches a dedicated subserver for each LLM, using delayed‑batching schedulers whose hyperparameters are derived from a throughput model. Across various routing algorithms, workloads, and model pairs, TokenRouter delivers 2.01‑64.15× higher decoding throughput than existing systems, markedly improving the efficiency of token‑level LLM inference.

arXiv AI

Oct 16

Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools

The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.

By Shivam Patel, Akaash R. Parthasarathy, Akash Dhasade, Ankur Mallick, Gauri Joshi

llms

arXiv AI

Jul 22

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

arXiv:2607. 18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost.

By Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick, Gauri Joshi

llms

arXiv AI

Oct 15

Stable Token-level Scheduling for Serverless LLM Inference

arXiv:2610.17689v1 Announce Type: new Abstract: Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, i...

By Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti Addya

llms efficiency

arXiv AI

Sep 1

Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference

arXiv:2608.28726v1 Announce Type: new Abstract: The remarkable performance of multimodal large language models (MLLMs) comes at the cost of substantial computational overhead, posing significant chal...

By Xinyuan Gui, Shaowen Wang, Sheng Sun, Zijian Wang, Zishu Yu, Zheming Yang

llms efficiency multimodal benchmarks

arXiv Machine Learning

Aug 11

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

arXiv:2510. 19366v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs).

By Xinfeng Xia, Xiaofeng Hou, Jiacheng Liu, Wenfeng Wang, Mingxuan Zhang, Peng Tang, Chao Li, Minyi Guo

llms efficiency

arXiv AI

Jul 7

Online Linear Programming for Multi-Objective Routing in LLM Serving

arXiv:2607. 03948v1 Announce Type: new Abstract: We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints.

By Zixi Chen, Yinyu Ye, Zijie Zhou

llms efficiency

arXiv AI

Jun 2

Lodestar: An Online-Learning LLM Inference Router

arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.

By Gangmuk Lim, Wanyu Zhao, Brighten Godfrey, Jiaxin Shan, Le Xu, Liguang Xie

llms benchmarks

arXiv Machine Learning

Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse

llms agents safety

arXiv AI

Jul 23

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.

By Tiancheng Zhang, Shaoyuan Huang, Mingyuan Wang, Yunfeng Zhao, Xiaofei Wang, Wenyu Wang

llms efficiency benchmarks

arXiv Machine Learning

Aug 21

RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

arXiv:2605. 23057v2 Announce Type: replace Abstract: RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference.

By Aman Sunesh, Ali Alshehhi, Hivansh Dhakne

llms efficiency benchmarks

arXiv Machine Learning

Aug 18

Multi-Bin Batching for Increasing LLM Inference Throughput

arXiv:2412. 04504v2 Announce Type: replace-cross Abstract: As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical.

By Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani

llms efficiency

arXiv Machine Learning

Sep 30

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

arXiv:2609.37062v1 Announce Type: cross Abstract: Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely...

By Wei Da, Yavuz Ferhatosmanoglu, Evangelia Kalyvianaki

llms efficiency

Related stories