← Back to all news
arXiv Machine Learning By Jack Cai, Lijie Yang, Vasilis Kontonis, Dimitris Papailiopoulos, Shivam Garg

AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens

Read the original on arXiv Machine Learning →

AbridgeKV compresses long chain‑of‑thought outputs by encoding completed text blocks into the KV cache of specially trained pause tokens, freeing the original cache space. The method uses pause tokens positioned within the block, allocates cache budget per layer based on entry value, and compresses large blocks. Experiments on 446 competition problems show that with only a quarter of the original KV cache, AbridgeKV matches the accuracy and output length of a full‑KV model, while enabling faster inference and a 3.5× smaller cache per sequence.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 7

Monte Carlo Estimation for KV Cache Eviction

arXiv:2610.07643v1 Announce Type: cross Abstract: Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter whi...

By Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer, Wajih Hassan Raza, Atta Ul Asad, Young D. Kwon, Michal Valko, Dean F. Hougen
llmsefficiency
More like this →
arXiv Machine Learning
Sep 14

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.

By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
llmsdiffusionefficiencysafety
More like this →
arXiv Machine Learning
Oct 9

When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression

The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.

By Haeyong Kang, Chang D. Yoo
More like this →