arXiv:2608. 06849v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
The paper introduces Pre-hoc Sparsity (PrHS), a method that selects key-value (KV) cache entries before attention scoring to avoid posterior bias in large language model inference. By bounding mutual‑information loss through the dropped attention mass, PrHS offers explicit accuracy control and implements three orthogonal selectors across time, depth, and layer. Experiments on LLaMA and Mistral models show that PrHS cuts retrieval overhead by over 90%, achieves higher sparsity than HShare, and delivers significant speedups and reduced FLOPs on NVIDIA A100 GPUs while maintaining near‑dense accuracy.
arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.
arXiv:2608. 02947v1 Announce Type: new Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position.
The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position. Aligned with this structure, we propose the per-RoPE-wavelength distance window: it prunes the query--key inner-product terms beyond a wavelength-proportional distance.