arXiv AI October 16, 2026 By Shivam Patel, Akaash R. Parthasarathy, Akash Dhasade, Ankur Mallick, Gauri Joshi

Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools

The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Jul 22

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

arXiv:2607. 18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost.

By Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick, Gauri Joshi

llms

arXiv Computation and Language

Oct 9

TokenRouter: Efficient Serving System for Token-Level LLM Routing

TokenRouter is a new serving system designed for token‑level routing of large language models (LLMs). It adopts a request‑centric programming model and launches a dedicated subserver for each LLM, using delayed‑batching schedulers whose hyperparameters are derived from a throughput model. Across various routing algorithms, workloads, and model pairs, TokenRouter delivers 2.01‑64.15× higher decoding throughput than existing systems, markedly improving the efficiency of token‑level LLM inference.

By Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang

llms efficiency

arXiv Machine Learning

Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse

llms agents safety

arXiv Machine Learning

Aug 11

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

arXiv:2510. 19366v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs).

By Xinfeng Xia, Xiaofeng Hou, Jiacheng Liu, Wenfeng Wang, Mingxuan Zhang, Peng Tang, Chao Li, Minyi Guo

llms efficiency

arXiv AI

Jun 2

Lodestar: An Online-Learning LLM Inference Router

arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.

By Gangmuk Lim, Wanyu Zhao, Brighten Godfrey, Jiaxin Shan, Le Xu, Liguang Xie

llms benchmarks

arXiv AI

Jul 23

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.

By Tiancheng Zhang, Shaoyuan Huang, Mingyuan Wang, Yunfeng Zhao, Xiaofei Wang, Wenyu Wang

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

Lodestar: An Online-Learning LLM Inference Router

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads