Learning the Cost of Reliable Inference
arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on t...
The paper proposes an inference auction for large language model (LLM) APIs, enabling users to bid for faster service when compute demand exceeds capacity. The auction allocates priority efficiently without increasing latency, and includes fast algorithms for truthful bidding and an autobidding agent that adjusts bids within a user’s budget to maximize utility. Experiments show the auction improves system welfare while preserving the cache utilization and latency benefits of the SGLang inference framework.
arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on t...
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.
arXiv:2606. 03092v1 Announce Type: new Abstract: Inference-time scaling has emerged as a critical avenue for enhancing Large Language Models' performance, yet real-world deployment is constrained by strict computational budgets.
arXiv:2607. 09600v1 Announce Type: new Abstract: Enhancing the reasoning capabilities of large language model (LLM) agents requires effective orchestration of diverse expert models and tools.
The paper introduces LLM‑Auction, a learning‑based generative auction mechanism that integrates ad allocation with large language model (LLM) generation. It redefines the auction object from discrete ad slots to distributions over LLM outputs, aligning allocation with advertiser value and user experience while inherently modeling allocation externalities without extra inference cost. The authors prove allocation monotonicity and continuity, show a first‑price payment rule has favorable incentive properties, and demonstrate state‑of‑the‑art allocation efficiency in a simulated LLM‑as‑a‑judge environment.
arXiv:2604. 05845v2 Announce Type: replace-cross Abstract: Auto-bidding services optimize real-time bidding strategies for advertisers under key performance indicator (KPI) constraints such as target return on investment and budget.
arXiv:2608. 03705v1 Announce Type: new Abstract: Real-time bidding (RTB) ad exchanges typically forward nearly all incoming requests to demand-side platforms (DSPs), even though only a small fraction receive bids.
arXiv:2608. 11840v1 Announce Type: cross Abstract: Growing demand for artificial intelligence (AI) inference services requires scalable infrastructure, yet centralized serving costs rise with demand.
arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.
HAServe introduces a horizon-aware scheduling and memory co-design framework for efficient large language model (LLM) serving. By modeling request admission and key-value (KV) retention as a unified problem using the HorizonSchema abstraction, it selects joint admission and KV-retention candidates that minimize predicted latency across varying prefix structures and arrival rates. Experiments on Qwen3-4B-Instruct-2507 and Llama2-13B with eight H100 GPUs demonstrate up to 5.90× P99 TTFT speedups, 1.44× throughput gains, and a 20.59 percentage point increase in KV-cache hit rate over the strongest baseline.
arXiv:2607. 03948v1 Announce Type: new Abstract: We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints.
arXiv:2607. 18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost.