arXiv AI

Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

arXiv:2608. 20316v1 Announce Type: new Abstract: Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost.

arXiv AI
Oct 27

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

The paper investigates how users should set the reasoning‑effort level in large language models, a parameter that trades off inference cost for potential performance gains. Experiments on proprietary and open‑weight models across four benchmarks show that higher effort does not consistently improve accuracy, can be up to 42× more expensive on easy tasks, and alters agent behavior by increasing verification and search steps. The authors propose a simple escalation strategy—starting at the lowest effort and only increasing after a verified failure—that matches or outperforms fixed high‑effort settings while reducing cost.

By Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou, Federico Bianchi
arXiv AI
Oct 2

OR for AI That Does OR: Routing LLMs up the Escalator inside the OSCAR Framework

The paper introduces OSCAR, an LLM‑based framework that translates business descriptions into accurate optimization models while verifying and improving them through a simulator, coder, and reviewer. OSCAR uses a cost‑ordered escalation strategy to select among LLMs of varying price and capability, achieving 95–100% accuracy on benchmark problems with local, open‑weight models. The framework also provides competitive guarantees and token‑cost advantages over existing LLMs like Codex and Claude Code.

By Jinzhi Bu, Haixin Tang, Huanan Zhang
arXiv AI
Sep 30

You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

The paper demonstrates that in open‑weight LLM inference markets, selecting a model is insufficient; clients must also choose a provider, as the same model can differ markedly in quality, latency, availability, and price across providers. The authors propose a market‑aware routing approach, including a measured‑map policy and an online router called FACET, which certifies provider feasibility for each task and safely falls back to a reliable anchor. Experiments show that this strategy yields cost savings while maintaining quality and avoiding degraded endpoints.

By Liang He, Jingbo Wen, Yixiong Chen, Yue Yang, Qizhen Lan, Kangning Cui, Xilu Wang
arXiv AI
Nov 6

When LLM Routers Overpay: The Cost of Strong-Model Preference

The paper examines how large language model (LLM) routers can overpay by favoring the strongest, most expensive models even when cheaper alternatives offer similar utility, especially under loose cost budgets. This strong-model over-selection is not solely due to ambiguity but arises when cost differences persist despite equivalent performance. To address this, the authors introduce EquiRouter, a cost‑aware routing method that learns query‑dependent rankings and explicitly prefers lower‑cost models when utilities are comparable, achieving better cost efficiency without sacrificing performance.

By Guannan Lai, Long Chen, Han-Jia Ye