arXiv Machine Learning
Jun 8
Towards Tight Bounds for Streaming Attention
arXiv:2606. 07205v1 Announce Type: cross Abstract: The attention mechanism is a cornerstone of modern transformer architectures.
Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
The paper introduces two new kernel‑based methods, GAIM and Q‑DAIM, for compressing the key‑value cache in attention mechanisms by formulating the problem in a Hilbert space and applying empirical interpolation techniques. To reduce the quadratic cost of these methods, the authors propose surrogate models—including Taylor series, Performer, and Random Fourier Features—that enable linear‑time compression. The work also extends to streaming attention, offering node‑selection penalties and interpolation updates, and demonstrates competitive performance on the LongBench‑E benchmark.
Related stories
arXiv Machine Learning
Aug 26
PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
arXiv AI
Jul 24
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
arXiv AI
Oct 16
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
DropKV introduces a new KV‑cache eviction framework that decouples residual‑output perturbation into independent per‑token scores, avoiding the NP‑hard combinatorial optimization of prior methods. The approach offers a provable constant‑factor approximation under a cone condition verified on three long‑context LLMs, and a high‑performance Triton‑based implementation that yields a 19.8× speedup in scoring kernels and up to 9.9% end‑to‑end prefill speedup. Benchmarks on RULER and LongBench show DropKV consistently outperforms state‑of‑the‑art baselines at the same cache budgets, achieving the lowest inference latency among eviction strategies while remaining faster than dynamic cache baselines.
arXiv Machine Learning
Oct 2
PatchKV: Weight-Space Compensation of KV Cache
arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
arXiv AI
Oct 1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
arXiv:2602.08005v2 Announce Type: replace-cross Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computati...
arXiv Machine Learning
Jun 2
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
arXiv AI
Jun 10
Blurry Window Attention
arXiv:2606. 09862v1 Announce Type: cross Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.
Hugging Face Trending Papers
Jun 23
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
Hugging Face Trending Papers
Oct 15
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
DropKV introduces a new key-value cache eviction strategy for large language models that decouples the joint eviction decision into independent per-token scoring, thereby avoiding the NP-hard combinatorial problem of minimizing output perturbation. The method offers a provable constant-factor approximation guarantee under a cone condition verified on three long-context LLMs, and its high-performance implementation using fused Triton kernels achieves a 19.8× speedup in scoring and up to 9.9% end-to-end prefill speedup. Evaluations on RULER and LongBench show that DropKV outperforms state-of-the-art baselines at equivalent cache budgets, delivering the lowest inference latency among eviction methods while remaining faster than the dynamic cache baseline.
arXiv AI
Jun 24
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv Machine Learning
Jul 28
DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
arXiv:2607. 24331v1 Announce Type: new Abstract: As the inference phase of Large Language Models (LLMs) requires handling long context windows, the Key-Value (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context window continues to grow.