Elastic Threshold Attention (ETA) is a trainable attention mechanism that dynamically predicts contextual thresholds from query representations, enabling selective pruning of KV cache tokens during long‑context decoding. By multiplicatively suppressing sub‑threshold logits during training, ETA avoids representation collapse and eliminates localized attention sinks, allowing a 1.45B model to match dense attention performance at roughly 85% training sparsity and 38% active decode density. At inference, a custom Triton kernel achieves up to 2.5× faster decoding on sequences up to 512K tokens, and an offline calibration step can further reduce compute by 27% by freezing per‑head thresholds.
By Themistoklis Haris, Henry Li, Maryam Karimzadehgan
CompKV introduces a compensation‑aware sparse attention framework for long‑context LLM inference. It partitions tokens into blocks and optimizes token selection to minimize the error introduced by block‑level mean compensation, using compact block‑level statistics. Experiments on RULER and LongBench‑Pro show CompKV outperforms other sparse baselines and achieves up to a 6.85× speedup over full attention.
By Zhen Huang, Ruizhe Yao, Danyi Liu, Xinrui Chen, Shuwei Li, Siru Zhong, Zijian Cao, Yushan Lai, Mingming Guo, Weijie Zheng, Haohuan Fu
Large language models (LLMs) often process entire input contexts even when only a small portion is needed to answer a query. The authors introduce dynamic context cutoff, a technique that lets LLMs self‑terminate processing once they have gathered enough task‑relevant information. By analyzing model internals, they find that certain attention heads encode latent sufficiency signals that can be detected with lightweight classifiers, enabling efficient context handling. Experiments on six QA datasets with LLaMA, Qwen, and Mistral models (1B–70B) show a 3.4 % accuracy gain and a 1.33× reduction in token usage, outperforming other context‑efficiency methods at similar token reductions. The study also notes that larger models can intrinsically assess sufficiency through prompting, while smaller models need explicit probing.
By Roy Xie, Junlin Wang, Paul Rosu, Chunyuan Deng, Bolun Sun, Zihao Lin, Bhuwan Dhingra
arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.
By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan
The paper introduces On‑Demand Attention (ODA), a decoding strategy that lets pretrained language models decide when to use global attention based on a lightweight recall head. ODA keeps the original model weights unchanged, only training the recall head, and can be implemented with GPU‑side conditional execution to reduce global reads. Experiments on Qwen, Gemma, and hybrid‑attention models show that ODA largely recovers performance lost by local attention while cutting the number of global attention operations.
By Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu
arXiv:2506.19549v2 Announce Type: replace-cross
Abstract: Identifying which tokens and activations truly influence model predictions is critical for both the efficiency and interpretability of large...
By Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena, Subrata Mitra