arXiv:2606. 18774v1 Announce Type: new Abstract: We present RouteJudge, an online pairwise preference evaluation framework for LLM routing systems, with a public platform available at https://routejudge.
Dynamic LLM routers aim to reduce inference costs by directing each query to the cheapest capable model. In a study of six commercial routers across 14 settings and eight task categories, none surpassed a simple random router that selects between two well-chosen models at the same cost, with some underperforming by over 10 percentage points. The authors identify four common patterns—difficulty blindness, length reversal, semantic matching, and roster suboptimality—that explain this gap and propose a new evaluation method and a simple two-model router that mitigates these patterns, though its advantage over random routing remains modest.
arXiv:2603.04445v3 Announce Type: replace-cross Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent mo...
The paper demonstrates that in open‑weight LLM inference markets, selecting a model is insufficient; clients must also choose a provider, as the same model can differ markedly in quality, latency, availability, and price across providers. The authors propose a market‑aware routing approach, including a measured‑map policy and an online router called FACET, which certifies provider feasibility for each task and safely falls back to a reliable anchor. Experiments show that this strategy yields cost savings while maintaining quality and avoiding degraded endpoints.
arXiv:2607. 23765v1 Announce Type: cross Abstract: Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale.
The paper introduces SaveRouter, a sparse‑supervision framework for large language model routing that selectively gathers informative model feedback and shares capability information across related queries. By using only about 33–41% of available training feedback, SaveRouter achieves competitive or superior routing quality while reducing the break‑even deployment volume by 1.9–9.5× compared to conventional routers. The study also shows that the supervision level that minimizes serving cost may differ from the one that yields the earliest payback.
arXiv:2606. 06178v1 Announce Type: new Abstract: Large language models (LLMs) present a trade-off between performance and cost, where more powerful models incur greater expense.
The paper investigates how performance predictors can improve model routing for large language models (LLMs). It shows that a predictor only outperforms a model’s self‑confidence when the candidate model is suboptimal or the predictor adds significant capacity. The authors provide theoretical bounds, extend the analysis to two‑model cascades, and confirm that predictors help most with suboptimal models, with gains diminishing after fine‑tuning.
The paper investigates how performance predictors can improve model routing for large language models (LLMs). It shows that a predictor only outperforms a model’s self‑confidence when the candidate model is suboptimal or the predictor adds significant capacity. The authors provide theoretical bounds, extend the analysis to two‑model cascades, and confirm their findings experimentally across 16 LLMs on four benchmarks, noting that the advantage diminishes as models are fine‑tuned.
arXiv:2606.27457v2 Announce Type: replace-cross Abstract: Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes...
arXiv:2606. 02581v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) faces a fundamental three-way tension: deeper retrieval improves factual grounding but inflates token costs and end-to-end latency.
arXiv:2609.37362v1 Announce Type: new Abstract: Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--eff...