Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
Read the original on arXiv Machine Learning →The paper introduces Prefix‑Certificate Retention (PCR), an exact optimization framework for deciding which prefix states of a language model to cache in order to balance recomputation and storage time. PCR models usable‑prefix rewards as nodes with prerequisites tied to timeout thresholds and preceding hit certificates, reducing the problem to a single minimum‑cut graph whose size scales linearly with block lookups and timeout choices. The authors provide a breakpoint theorem that extends the construction to all nonnegative timeouts without discretization error, a linear‑time dynamic program for ordered timeouts, and empirical validation on 39,632 public Mooncake requests, showing that ordered timeouts achieve the unrestricted optimum in most trace‑grouping cases and that heterogeneous retention can improve memory‑time tradeoffs. "whyItMatters":"The study offers a tractable, auditable optimization model for retention policies that directly measures usable prefix blocks and storage time, providing a concrete benchmark for evaluating memory‑time tradeoffs in language‑model serving."
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.