The paper investigates the use of partial Rotary Positional Embedding (RoPE) in transformer models, showing that applying RoPE to only about 10% of hidden dimensions can save up to ten times the memory of full RoPE while achieving comparable final loss. The authors systematically analyze training dynamics and convergence across various architectures, model sizes, sequence lengths, and datasets, finding consistent performance trends. They also note that completely omitting positional encoding (NoPE) can cause loss spikes, which are mitigated by minimal RoPE usage or QK‑Norm application.
arXiv:2602. 02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache.
arXiv:2607. 10134v1 Announce Type: new Abstract: Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models.
arXiv:2608.29715v1 Announce Type: cross Abstract: Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional informa...
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv:2610.06963v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific fr...
Kronecker Product Attention (KPA) is introduced as a memory‑efficient method for compressing the key‑value cache in autoregressive models. By representing key and value activations as sums of Kronecker products of compact, token‑dependent matrix factors, KPA generalizes Tensor Product Attention and can capture richer block‑structured patterns. Empirical results show that KPA matches or surpasses existing techniques such as MHA, MQA, GQA, and MLA across multiple model scales and benchmarks, achieving up to 10× compression while preserving relative‑position information through a structured rotary embedding.
arXiv:2511. 17388v3 Announce Type: replace-cross Abstract: Position information is essential for language modeling.
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
arXiv:2607.12550v3 Announce Type: replace-cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, a...
The paper investigates how the positional geometry of cached key‑value (KV) states affects inference quality in sparse KV compression. It introduces RoPERelay, a method that reassigns compact, order‑preserving RoPE coordinates and re‑rotates cached key phases to better align with current queries, without requiring new selection or extra cache budget. Experiments on two models and multiple selectors show that RoPERelay improves answer quality in most cases, notably raising Llama‑3.1‑8B‑Instruct accuracy from 23.6% to 35.4% on LongMemEval‑S with only a 1–3% runtime overhead.
arXiv:2607. 20538v1 Announce Type: cross Abstract: Long-context Transformer inference increasingly relies on KV-cache compression or quantization.
arXiv:2607. 15456v1 Announce Type: new Abstract: Looped, weight-tied Transformers reduce parameters by reusing a block, but decoding still stores a separate K/V cache for every recurrence step.