The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
arXiv:2608. 15810v1 Announce Type: new Abstract: Runtime compression of serving state trades quality for capacity with no priced guarantee: systems adapt precision on load signals with no soundness statement, and certified approaches budget request-level risk by a union bound over a pre-declared event count.
arXiv:2608. 05863v1 Announce Type: new Abstract: Modern models no longer keep a plain KV cache: latent caches, learned sparse selectors and recurrent states each carry the model's memory in a different form, and each fails differently under compression.
The paper argues that large language model (LLM) providers, constrained by compute, often degrade service during congestion by routing queries to smaller models, cutting reasoning effort, or truncating context. It shows that this practice misrepresents costs because degraded answers can fail, leading to retries that inflate traffic or churn that erodes lifetime value. By modeling inference allocation with newsvendor, retry, and queueing frameworks, the authors derive a ‘shadow price of intelligence’ that quantifies the marginal value of each query, revealing that throttling under congestion acts as a demand lever rather than a cost lever.
arXiv:2607. 16891v1 Announce Type: cross Abstract: A truckload carrier must accept or reject each load tender within seconds.
arXiv:2606. 07846v1 Announce Type: cross Abstract: LLM-agent workflows chain model calls and tool invocations, and spend most of their wall-clock time waiting on upstream operations before downstream ones can start.
arXiv:2608. 07911v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard.