arXiv Machine Learning
Pathwise-Budgeted LLM Routing: Online Allocation with K-Bit Tail-Cost Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 28
WISERouter: LLM Routing with Workload Budget Constraint
arXiv:2607. 23765v1 Announce Type: cross Abstract: Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale.
arXiv AI
Sep 2
Drift-Aware LLM Routing with Sparse Contexts and Shared Budgets
The paper introduces Drift‑Aware Sparse Routing (DRS), a method for routing requests in a multi‑model language service while respecting compute, latency, memory, or cost budgets. DRS estimates reward and resource use from a rolling audit window, routes using pessimistic reward and optimistic cost estimates, updates resource shadow prices online, and applies a hard meter before commitment. The authors provide theoretical regret bounds that separate control from statistics, showing how the method adapts to non‑stationary prompt distributions and model changes.
arXiv AI
Oct 15
Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing
arXiv:2610.15993v1 Announce Type: new Abstract: We study how initial model selection, later handoff, and training targets affect success and cost in LLM tasks that span multiple turns. We compare sev...
arXiv Machine Learning
Oct 9
Amortized Off-Policy Evaluation for LLMs
The paper introduces PFN‑OPE, a prior‑data fitted network that performs amortized off‑policy evaluation (OPE) for large language models (LLMs). Unlike traditional OPE methods that must be retrained for each new logged dataset or reward definition, PFN‑OPE is pretrained once on a diverse set of contextual‑bandit tasks and can estimate value for any new logged dataset and target response in a single forward pass. Experiments on HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies show that PFN‑OPE reduces error by 2.0 to 9.3 times compared to the best baselines in reward‑shifted settings.
arXiv AI
Sep 2
Bandits in Prod: Hyperparameter Optimization at Inference Time
The paper introduces Online Hyperparameter Optimization (OHPO), framing it as an infinitely many‑armed bandit problem over mixed and conditional search spaces. It proposes the IMABO framework, which couples any bandit policy with any oracle for proposing new configurations, and presents IMOSS—a restart‑free anytime policy with provable regret bounds. Experiments show that IMABO, combined with practical oracles such as TPE, an incumbent‑mutation oracle, and a pretrained tabular foundation model, outperforms random search across a range of settings from classical ML models to LLM‑based agents.
arXiv Machine Learning
Jun 5
Cross-Epoch Adaptive Rollout Optimization for RL Post-Training
arXiv:2606. 05606v1 Announce Type: new Abstract: LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide.