← Back to all news
arXiv Computation and Language By Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks, Yaniv Galron, Talor Abramovich, Yoav Miron, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein

RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

Read the original on arXiv Computation and Language →

RIFT introduces selection‑based KV cache compression strategies that balance prefill information with recently generated tokens to mitigate decode staleness in speculative decoding. By refreshing the KV cache selection during token generation and employing an adaptive refresh strategy, RIFT improves drafter quality while managing decode latency. A compressibility‑driven drafter training phase further enhances performance, enabling Qwen3.5‑4B to retain over 98% of its full‑cache acceptance length within an 8K‑token KV cache budget on long‑context inputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

Hugging Face Trending Papers
Oct 22

RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

The paper introduces RIFT, a set of selection‑based KV cache compression strategies that balance prefill information with recently generated tokens to mitigate decode staleness in draft models. RIFT improves drafter quality by refreshing KV cache selection during token generation, and an adaptive refresh strategy reduces score calculations to lower latency. Additionally, a compressibility‑driven drafter training phase using sparse attention masks further enhances performance, enabling Qwen3.5‑4B to maintain over 98% of its full‑cache acceptance length within an 8K‑token KV cache budget on long‑context inputs.

efficiency
More like this →
arXiv Computation and Language
Sep 10

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.

By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi
ragfine-tuningefficiencybenchmarks
More like this →