The paper investigates how performance predictors can improve model routing for large language models (LLMs). It shows that a predictor only outperforms a model’s self‑confidence when the candidate model is suboptimal or the predictor adds significant capacity. The authors provide theoretical bounds, extend the analysis to two‑model cascades, and confirm their findings experimentally across 16 LLMs on four benchmarks, noting that the advantage diminishes as models are fine‑tuned.
By Guanhua Zhang, Moritz Hardt
The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.
By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana
The paper introduces a unified framework called cascade routing that combines routing and cascading strategies for selecting large language models (LLMs) in agentic systems. It derives an optimal cascading strategy, proves the optimality of an existing routing strategy, and demonstrates that integrating both leads to a theoretically optimal approach. Experiments show cascade routing consistently outperforms the individual methods, and the study identifies high‑quality estimators as crucial for effective model selection.
By Jasper Dekoninck, Maximilian Baader, Martin Vechev
arXiv:2609.38006v1 Announce Type: new
Abstract: Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable tha...
By Kenan Alkiek, Moontae Lee, David Jurgens, V. G. Vinod Vydiswaran
arXiv:2603.04445v3 Announce Type: replace-cross
Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent mo...
By Yasmin Moslem, John D. Kelleher
arXiv:2602.03478v2 Announce Type: replace
Abstract: LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they...
By Guannan Lai, Long Chen, Han-Jia Ye