arXiv AI
GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 14
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
arXiv AI
Sep 1
Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
arXiv:2608.30310v1 Announce Type: cross Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
arXiv Machine Learning
Sep 24
DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video
DeltaS is a query‑agnostic, training‑free method for evicting key‑value cache entries in hybrid video‑language models that combine linear and full attention. It uses the change in the recurrent state of gated‑delta linear attention—called state drift—to decide which video chunks to keep, selecting those that induce larger normalized state changes. In experiments with a fixed memory budget, DeltaS outperforms position‑, attention‑, and key‑value‑based eviction signals, improving performance by 2.1 points on average across six long‑video benchmarks and 5.6 points on the longest benchmark, while adding only 1.9% of the forward‑pass cost.
arXiv AI
Oct 7
Mask-Guided KV Cache Eviction in Block Diffusion Language Models
arXiv:2610.06996v1 Announce Type: cross Abstract: Block diffusion language models keep a large key-value (KV) cache throughout generation and attend to it at every denoising step, limiting both memor...
arXiv Machine Learning
Sep 22
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
arXiv Machine Learning
Oct 8
KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...