arXiv Machine Learning

ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing

arXiv:2607. 15899v1 Announce Type: new Abstract: In production large language model (LLM) deployments, high API availability guarantees do not equate to conversational continuity.

arXiv Machine Learning

Aug 12

Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy

arXiv:2608. 10532v1 Announce Type: cross Abstract: Static load balancers cannot mitigate a backend that is degraded rather than down: round-robin and least-connections keep routing traffic to a server returning HTTP 500s until an operator intervenes.

By Aman Chauhan, Vishnu Pendyala

llms benchmarks safety

arXiv Machine Learning

Aug 18

Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving

arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.

By Anders Vestrum, Arya Raeesi, Hanna Roed

llms

arXiv AI

Jul 21

Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

arXiv:2607. 17181v1 Announce Type: cross Abstract: Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently.

By Utopia Meng, Unicornt Zhao, Derek Li, Goalen Gao, Frank Du

llms agents

arXiv AI

Jul 23

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.

By Tiancheng Zhang, Shaoyuan Huang, Mingyuan Wang, Yunfeng Zhao, Xiaofei Wang, Wenyu Wang

llms efficiency benchmarks

arXiv AI

Jun 3

ProtocolBench: Which LLM MultiAgent Protocol to Choose?

arXiv:2510. 17149v3 Announce Type: replace Abstract: As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability.

By Hongyi Du, Jiaqi Su, Jisen Li, Lijie Ding, Yingxuan Yang, Peixuan Han, Xiangru Tang, Kunlun Zhu, Jiaxuan You

llms agents benchmarks

arXiv AI

Jun 16

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

arXiv:2605. 21312v2 Announce Type: replace-cross Abstract: Modern LLM serving is no longer homogeneous or monolithic.

By Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang, Yibo Zhu, Hong Xu

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse

llms agents safety

arXiv AI

Sep 23

From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving

arXiv:2609.23130v1 Announce Type: new Abstract: Large language model (LLM) inference is evolving from an engine-local optimization problem into a distributed control problem involving reusable state,...

By Twinkll Sisodia

llms agents efficiency multimodal benchmarks

arXiv AI

Jul 20

Scalable LLM Agent Tool Access in the Cloud

arXiv:2607. 15593v1 Announce Type: cross Abstract: LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface.

By Mingxin Li, Enge Song, Yueshang Zuo, Xiaodong Liu, Rong Wen, Qiang Fu, Gianni Antichi, Jian He, Jing Tie, Zhou Shao, Xiaobo Xue, Xiong Xiao, Luyao Zhong, Shaokai Zhang, Jiangu Zhao, Jianyuan Lu, Shize Zhang, Xiaoqing Sun, Changgang Zheng, Zihao Fan, Haonan Li, Tian Pan, Xiaomin Wu, Yang Song, Xing Li, Biao Lyu, Meng Li, Haipeng Dai, Guihai Chen, Shunmin Zhu

llms agents efficiency

arXiv AI

4d ago

TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing

arXiv:2605.18859v3 Announce Type: replace-cross Abstract: LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single u...

By Pei Yang, Wanyi Chen, Tongyun Yang, Pengbin Feng, Jiarong Xing, Wentao Guo, Yuhang Yao, Yuhang Han, Hanchen Li, Xu Wang, Zeyu Wang, Jie Xiao, Anjie Yang, Liang Tian, Lynn Ai, Eric Yang, Tianyu Shi

llms agents benchmarks

arXiv Computation and Language

Sep 23

Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

arXiv:2609.26693v1 Announce Type: new Abstract: A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action....

By Lijuan Tang, Yuemeng Zheng

llms agents

arXiv AI

Aug 12

Conversational Orchestration for Organic 6G

arXiv:2608. 10714v1 Announce Type: cross Abstract: The Organic 6G vision of a network of networks spanning an edge-cloud continuum complemented by non-terrestrial resources requires, to realize its promise, service provisioning that is simple to operate, scalable across independently administered domains, and agile under domain churn (i.

By Masoud Shokrnezhad, Tarik Taleb

llms agents

Related stories