arXiv AI By Liang He, Jingbo Wen, Yixiong Chen, Yue Yang, Qizhen Lan, Kangning Cui, Xilu Wang

You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference

Read the original on arXiv AI →

The paper demonstrates that in open‑weight LLM inference markets, selecting a model is insufficient; clients must also choose a provider, as the same model can differ markedly in quality, latency, availability, and price across providers. The authors propose a market‑aware routing approach, including a measured‑map policy and an online router called FACET, which certifies provider feasibility for each task and safely falls back to a reliable anchor. Experiments show that this strategy yields cost savings while maintaining quality and avoiding degraded endpoints.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Nov 6

When LLM Routers Overpay: The Cost of Strong-Model Preference

The paper examines how large language model (LLM) routers can overpay by favoring the strongest, most expensive models even when cheaper alternatives offer similar utility, especially under loose cost budgets. This strong-model over-selection is not solely due to ambiguity but arises when cost differences persist despite equivalent performance. To address this, the authors introduce EquiRouter, a cost‑aware routing method that learns query‑dependent rankings and explicitly prefers lower‑cost models when utilities are comparable, achieving better cost efficiency without sacrificing performance.

By Guannan Lai, Long Chen, Han-Jia Ye
arXiv AI
Sep 30

TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing

arXiv:2605.18859v3 Announce Type: replace-cross Abstract: LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single u...

By Pei Yang, Wanyi Chen, Tongyun Yang, Pengbin Feng, Jiarong Xing, Wentao Guo, Yuhang Yao, Yuhang Han, Hanchen Li, Xu Wang, Zeyu Wang, Jie Xiao, Anjie Yang, Liang Tian, Lynn Ai, Eric Yang, Tianyu Shi
arXiv Computation and Language
Oct 26

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

arXiv:2606.27457v2 Announce Type: replace-cross Abstract: Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes...

By Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas, Solmaz Panahi, David Lynch, Zhuangzhuang Nie, Alexandros Agapitos, Aleksandar Milenovic, Hongmeng Song, Yucheng Shi, Yue Pan, Patricia Buffini, John D. Kelleher
arXiv AI
Sep 24

Learning the Cost of Reliable Inference

arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on t...

By Dimitrios Rontogiannis, Ander Artola Velasco, Manuel Gomez Rodriguez
arXiv Machine Learning
Sep 29

HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference

HybridInfer is a thermal‑aware reinforcement‑learning router that selects among on‑device, edge, and cloud large language model tiers based on a phone’s thermal headroom and query complexity. Trained offline with a Q‑learning policy, it balances quality, latency, cost, and a thermal penalty, and includes a locality bonus that encourages on‑device execution. In real Android tests on 210 prompts, the learned router outperformed two hand‑tuned heuristics in quality while maintaining the lowest cost, and it proved more reliable and faster than always‑on‑device inference, especially for long queries.

By Simran Koul