arXiv:2605. 04893v2 Announce Type: replace Abstract: When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal.
By Dominik Dahlem, Diego Maniloff, Mac Misiura
TwinKV is a training‑free, attention‑free repair pass that identifies and swaps orphaned and redundant tokens in a KV cache, improving long‑context inference for small models. It works by detecting near‑duplicate keys and can be composed with existing eviction policies without altering their scoring rules. Experiments on Qwen3‑4B and Llama‑3.2‑1B across LongBench, LooGLE, RULER, and MMLU‑Pro show that TwinKV consistently improves performance for most configurations, especially at tighter compression ratios.
By Hong Chen, Yudong Zeng, Yongwei Huang, Zuhao Ouyang, Junyan Zhang, Xuming Hu
arXiv:2607. 19368v1 Announce Type: new Abstract: Long-prompt inference remains expensive because prefill attention scales quadratically with sequence length.
By Ali Mahdavi, Azaseh Zamanifar, Amirfarhad Farhadi, Omid Kashefi
BF1 is a deterministic block‑aligned dyadic sparse‑attention retrofit designed to reduce the cost of causal attention in long‑context transformers. It combines a small exact local neighborhood, a global first block, and logarithmically spaced historical blocks, achieving O(n log n) token interactions per layer with O(log n) communication depth. On an NVIDIA RTX PRO 6000 Blackwell GPU, BF1 outperforms dense attention for 2K–4K tokens and delivers up to a 10.91× prefill speedup at 32K tokens, while retrofitting eight of 28 Qwen3‑0.6B layers reduces first‑token latency by up to 15.3% at 32K tokens and yields the lowest perplexity among compared sparse and dense training protocols.
By Hina Dixit
arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.
By Sagar Dangal, Manoj Shakya
arXiv:2511. 10696v3 Announce Type: replace-cross Abstract: Sparse attention is crucial in long-context Transformers, which restricts each token to a limited neighborhood and thereby reduces the quadratic cost of full self-attention.
By Pike D. Liu, Chang Liu, Yanxuan Yu