arXiv AI
Learning the Cost of Reliable Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 3
The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs
arXiv:2606. 03092v1 Announce Type: new Abstract: Inference-time scaling has emerged as a critical avenue for enhancing Large Language Models' performance, yet real-world deployment is constrained by strict computational budgets.
arXiv AI
1d ago
Inference Auctions
The paper proposes an inference auction for large language model (LLM) APIs, enabling users to bid for faster service when compute demand exceeds capacity. The auction allocates priority efficiently without increasing latency, and includes fast algorithms for truthful bidding and an autobidding agent that adjusts bids within a user’s budget to maximize utility. Experiments show the auction improves system welfare while preserving the cache utilization and latency benefits of the SGLang inference framework.
arXiv AI
Aug 14
Keep, Customize, or Exit: Default Design and Token Pricing in LLM Reasoning Services
arXiv:2608. 13315v1 Announce Type: cross Abstract: We study a large language model (LLM) service in which a provider chooses a per-token price and a default reasoning-token allocation, while a user may accept the default, customize the allocation, or exit.
Hugging Face Trending Papers
Aug 13
Keep, Customize, or Exit: Default Design and Token Pricing in LLM Reasoning Services
We study a large language model (LLM) service in which a provider chooses a per-token price and a default reasoning-token allocation, while a user may accept the default, customize the allocation, or exit. Larger allocations can improve accuracy but increase token cost and latency.
arXiv AI
Jun 19
Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees
arXiv:2606. 19376v1 Announce Type: cross Abstract: Inference costs for large language model (LLM) applications are rapidly growing, driven by surging demand and rising infrastructure cost.
arXiv AI
Aug 17
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
arXiv:2608. 13571v1 Announce Type: cross Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time.