← Back to all news
arXiv Computation and Language By Utkarsh Kanwat

Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels

Read the original on arXiv Computation and Language →

The paper introduces a method to incorporate attention bias directly into the keys of compacted KV caches, enabling standard kernels like FlashAttention and vLLM to use these folded caches without modification. By fitting a bias along a closed‑form direction per head, the approach preserves the cache size and head dimension while bounding output error based on query projections. Experiments on Qwen3 and Llama‑3.1 models show that the folded caches match or exceed the accuracy of the original Attention Matching method across various benchmarks, with comparable or lower KL divergence to full caches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 27

Trust the Mass: Forced Weights in KV-Cache Eviction

The paper investigates KV‑cache eviction strategies for sparse‑attention models, showing that selecting the largest attention weights is nearly optimal—closing only a median 2–5 % of the gap to full attention. It further demonstrates that differences in performance between eviction methods largely stem from memory usage, with the new training‑free ContourKV allocator outperforming state‑of‑the‑art methods in most pairwise comparisons while matching their byte‑efficiency.

By Jack Shi, Jerry Gu
benchmarks
More like this →
arXiv Machine Learning
Sep 4

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Eviction Signal in a Vestigial Branch

VestigeKV is a new KV‑cache technique that uses a 64‑dimensional vestigial branch—originally a RoPE component repurposed during NoPE training—as a query‑independent eviction signal. By reading only 11 % of each cache row, the method partitions the cache into an attended tier (top‑m rows) and an archive tier (all other rows), which is GPU‑resident and never deleted. The approach achieves near‑perfect retrieval (1.00 at 8×, 0.92 at 32×) without any training, quantization, or changes to weights or kernels.

By WenJie Fan
efficiency
More like this →
arXiv Machine Learning
Sep 22

ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs

ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.

By Junyoung Park, Jungwook Choi, Mingu Lee
llmsefficiencybenchmarks
More like this →
arXiv Computation and Language
Sep 7

Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation

The paper investigates how to recover language model quality lost when using low‑bit key–value caches for autoregressive decoding. By keeping the quantizer fixed and distilling the full‑precision cache behavior into low‑rank Q/K/V projection updates, the authors demonstrate that 4‑bit affine‑cache adapters recover roughly 54 % of the perplexity gap on TinyLlama‑1.1B and 76 % on Gemma‑4‑12B, while preserving most long‑context retrieval. Even a 2‑bit rank–token sweep can dramatically reduce TinyLlama’s perplexity, though it only partially restores retrieval performance.

By Seifeldin Abdellatif
llmsefficiency
More like this →