arXiv Machine Learning October 8, 2026 By Davis Wertheimer, Haochen Shen, Ahan Gupta, Derrick Liu, Yu Chin Fabian Lim, Mudhakar Srivatsa, Raghu K. Ganti, Minjia Zhang, Naigang Wang

A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention

Read the original on arXiv Machine Learning →

The paper introduces Universal Attention, a transformer architecture that unifies and extends decay-based KV‑cache pruning mechanisms while preserving RoPE embeddings and Softmax attention. It achieves highly expressive, end‑to‑end trainable pruning that adaptively removes low‑contribution tokens, yielding up to 10× compression on standard datasets and 25× compression at 16k context length without sacrificing downstream performance. The approach outperforms existing state‑of‑the‑art baselines and unpruned models in both compression and generalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 29

GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

arXiv:2607. 24788v1 Announce Type: new Abstract: As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck.

By Vimal William, Ravi Tandon, Jyotikrishna Dass

llms efficiency

arXiv AI

Jul 21

SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.

By Soumia Bouyahiaoui, Manel Kara laouar, Aicha Boutorh, Mohamed Hadj Ameur

llms efficiency safety

arXiv Computation and Language

Aug 25

Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression

arXiv:2606.14782v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning but incur large KV caches and high decoding latency with lo...

By Tianhao Chen, Yuheng Wu, Kelu Yao, Xiaogang Xu, Xiaobin Hu, Dongman Lee

llms efficiency multimodal benchmarks

arXiv AI

Jul 7

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.

By Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Yike Guo, Sirui Han

llms efficiency

arXiv AI

Jul 7

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

arXiv:2607. 01237v2 Announce Type: replace-cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.

By Shen Han, Yuyang Wu, Junpu Yu, Olexandr Isayev

llms efficiency

arXiv Machine Learning

Aug 17

KV Cache Compression Through the Lens of Transform Coding

arXiv:2608. 14191v1 Announce Type: new Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference.

By Hannah Laus, Claudio Mayrink Verdun, Hao Wang, Flavio du Pin Calmon, Felix Krahmer

llms efficiency benchmarks

One email a morning, machine-written

Related stories

GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression

IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference

KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

KV Cache Compression Through the Lens of Transform Coding