← Back to all news
arXiv Machine Learning By Michel Fabrice Serret, Laura Grigori

Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms

Read the original on arXiv Machine Learning →

The paper introduces two new kernel‑based methods, GAIM and Q‑DAIM, for compressing the key‑value cache in attention mechanisms by formulating the problem in a Hilbert space and applying empirical interpolation techniques. To reduce the quadratic cost of these methods, the authors propose surrogate models—including Taylor series, Performer, and Random Fourier Features—that enable linear‑time compression. The work also extends to streaming attention, offering node‑selection penalties and interpolation updates, and demonstrates competitive performance on the LongBench‑E benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 16

DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction

DropKV introduces a new KV‑cache eviction framework that decouples residual‑output perturbation into independent per‑token scores, avoiding the NP‑hard combinatorial optimization of prior methods. The approach offers a provable constant‑factor approximation under a cone condition verified on three long‑context LLMs, and a high‑performance Triton‑based implementation that yields a 19.8× speedup in scoring kernels and up to 9.9% end‑to‑end prefill speedup. Benchmarks on RULER and LongBench show DropKV consistently outperforms state‑of‑the‑art baselines at the same cache budgets, achieving the lowest inference latency among eviction strategies while remaining faster than dynamic cache baselines.

By Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Penghang Yin
llmsefficiencybenchmarks
More like this →