arXiv AI

Inference Auctions

The paper proposes an inference auction for large language model (LLM) APIs, enabling users to bid for faster service when compute demand exceeds capacity. The auction allocates priority efficiently without increasing latency, and includes fast algorithms for truthful bidding and an autobidding agent that adjusts bids within a user’s budget to maximize utility. Experiments show the auction improves system welfare while preserving the cache utilization and latency benefits of the SGLang inference framework.

arXiv AI
Sep 24

Learning the Cost of Reliable Inference

arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on t...

By Dimitrios Rontogiannis, Ander Artola Velasco, Manuel Gomez Rodriguez
arXiv Machine Learning
Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse
arXiv Machine Learning
Oct 23

LLM-Auction: Generative Auction towards LLM-Native Advertising

The paper introduces LLM‑Auction, a learning‑based generative auction mechanism that integrates ad allocation with large language model (LLM) generation. It redefines the auction object from discrete ad slots to distributions over LLM outputs, aligning allocation with advertiser value and user experience while inherently modeling allocation externalities without extra inference cost. The authors prove allocation monotonicity and continuity, show a first‑price payment rule has favorable incentive properties, and demonstrate state‑of‑the‑art allocation efficiency in a simulated LLM‑as‑a‑judge environment.

By Chujie Zhao, Qun Hu, Shiping Song, Dagui Chen, Han Zhu, Jian Xu, Bo Zheng
arXiv Machine Learning
Jul 22

JD-BP: A Joint-Decision Generative Framework for Auto-Bidding and Pricing

arXiv:2604. 05845v2 Announce Type: replace-cross Abstract: Auto-bidding services optimize real-time bidding strategies for advertisers under key performance indicator (KPI) constraints such as target return on investment and budget.

By Linghui Meng, Chun Gan, Shengsheng Niu, Chengcheng Zhang, Chenchen Li, Chuan Yang, Yi Mao, Xin Zhu, Jie He, Zhangang Lin, Ching Law
arXiv Machine Learning
Oct 16

HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

HAServe introduces a horizon-aware scheduling and memory co-design framework for efficient large language model (LLM) serving. By modeling request admission and key-value (KV) retention as a unified problem using the HorizonSchema abstraction, it selects joint admission and KV-retention candidates that minimize predicted latency across varying prefix structures and arrival rates. Experiments on Qwen3-4B-Instruct-2507 and Llama2-13B with eight H100 GPUs demonstrate up to 5.90× P99 TTFT speedups, 1.44× throughput gains, and a 20.59 percentage point increase in KV-cache hit rate over the strongest baseline.

By Xingyu Qu, Tianhao Lin, Yiqi Li, Zhiyu Chen, Sheng Wang