← Back to all news
arXiv Machine Learning By Guanhua Zhang, Moritz Hardt

Predictable performance is suboptimal, and what it means for model routing

Read the original on arXiv Machine Learning →

The paper investigates how performance predictors can improve model routing for large language models (LLMs). It shows that a predictor only outperforms a model’s self‑confidence when the candidate model is suboptimal or the predictor adds significant capacity. The authors provide theoretical bounds, extend the analysis to two‑model cascades, and confirm their findings experimentally across 16 LLMs on four benchmarks, noting that the advantage diminishes as models are fine‑tuned.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Oct 22

Predictable performance is suboptimal, and what it means for model routing

The paper investigates how performance predictors can improve model routing for large language models (LLMs). It shows that a predictor only outperforms a model’s self‑confidence when the candidate model is suboptimal or the predictor adds significant capacity. The authors provide theoretical bounds, extend the analysis to two‑model cascades, and confirm that predictors help most with suboptimal models, with gains diminishing after fine‑tuning.

llmsfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Oct 19

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.

By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana
llmsbenchmarks
More like this →
arXiv Computation and Language
Oct 28

A Unified Approach to Routing and Cascading for LLMs

The paper introduces a unified framework called cascade routing that combines routing and cascading strategies for selecting large language models (LLMs) in agentic systems. It derives an optimal cascading strategy, proves the optimality of an existing routing strategy, and demonstrates that integrating both leads to a theoretically optimal approach. Experiments show cascade routing consistently outperforms the individual methods, and the study identifies high‑quality estimators as crucial for effective model selection.

By Jasper Dekoninck, Maximilian Baader, Martin Vechev
llmsagents
More like this →