The paper introduces RAM‑Net, a mechanism that unifies KV‑cache quantization and linear attention by using soft assignments over a discrete address space to update continuous slot states. It shows that these soft assignments extend hard quantized matching to a separable read‑write overlap, locally approximating full‑attention similarity while enabling recurrent aggregation. The authors demonstrate that RAM‑Net can migrate pretrained Transformer weights and recover an average of 87.1% of the original accuracy across nine models using only a 500M‑token budget.
arXiv:2609.36760v1 Announce Type: new Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...
The paper introduces CORE, a method that jointly calibrates coverage and redistributes evicted mass for key–value (KV) cache in long‑context decoding. CORE combines query utility and log‑determinant coverage into a lightweight indexer, using a single calibrated distribution to both retain complementary KV states and parameterize latent‑memory writes. Experiments on three backbones show CORE outperforms the RULER baseline by up to 3.78 points at 90% compression and demonstrates strong decoding efficiency on LongBench and repeated‑eviction tests.
arXiv:2610.02235v1 Announce Type: cross Abstract: Long-context decoding is increasingly constrained by key--value (KV) cache memory and bandwidth. Existing fixed-budget compression methods typically...
arXiv:2608. 14191v1 Announce Type: new Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference.
arXiv:2602.08005v2 Announce Type: replace-cross Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computati...
arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.
The paper introduces Universal Attention, a transformer architecture that unifies and extends decay-based KV‑cache pruning mechanisms while preserving RoPE embeddings and Softmax attention. It achieves highly expressive, end‑to‑end trainable pruning that adaptively removes low‑contribution tokens, yielding up to 10× compression on standard datasets and 25× compression at 16k context length without sacrificing downstream performance. The approach outperforms existing state‑of‑the‑art baselines and unpruned models in both compression and generalization.
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.
arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.