arXiv AI
Mask-Guided KV Cache Eviction in Block Diffusion Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 7
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
arXiv Machine Learning
Sep 14
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
arXiv Machine Learning
Sep 30
EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
arXiv AI
Jun 30
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
arXiv Machine Learning
Jun 8
MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM
arXiv:2602. 14209v2 Announce Type: replace Abstract: Block diffusion LLMs are an emerging paradigm for parallel language generation, but their KV caching makes memory access the dominant bottleneck in long-context inference.
arXiv Machine Learning
Sep 22
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.