TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

TIDE (Temporal Incremental Draft Engine) is a serving‑engine‑native framework that integrates online draft adaptation into high‑performance LLM inference. By reusing intermediate hidden states from the target model as training signals, TIDE avoids extra target model computation and serving‑time overhead, activating speculation and draft training only when beneficial. On heterogeneous GPU clusters, TIDE achieves up to 1.66× higher throughput than no‑speculation baselines, reduces training time by up to 3.02×, cuts storage needs by 24×, and improves system throughput by up to 1.22×.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• efficiency