arXiv Machine Learning
Jun 24
Selective Rotary Position Embedding
arXiv:2511. 17388v3 Announce Type: replace-cross Abstract: Position information is essential for language modeling.
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
arXiv:2607. 10134v1 Announce Type: new Abstract: Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models.
Related stories
arXiv Machine Learning
Aug 4
RAP: KV-Cache Compression via RoPE-Aligned Pruning
arXiv:2602. 02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache.
arXiv AI
Jun 9
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
arXiv:2509. 10534v3 Announce Type: replace-cross Abstract: The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where.
arXiv Machine Learning
Jun 25
Why Do Accumulated Transformations Extrapolate?
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
arXiv AI
Jul 24
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
arXiv AI
Jul 23
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv Machine Learning
Sep 10
Content-Based Addressing for Long Context
The paper proposes a content‑based addressing scheme for long‑context models that replaces the growing token counter in Rotary Position Embedding (RoPE) with unit‑level addresses derived from the content of each unit. By dividing the token stream into units, the method preserves local RoPE behavior while allowing new units to be addressed via learned content maps, avoiding positional mismatches when extending context length. Experiments on character‑level Tiny Shakespeare show that a model trained on 256‑character contexts achieves lower perplexity at 4096 characters using this scheme, and a second diagnostic demonstrates retrieval of multiple serialized facts.
arXiv AI
6d ago
T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendation
T‑RoPE introduces a time‑aware Rotary Position Embedding for sequential recommendation, replacing index‑only rotations with timestamp‑based angles, learnable temporal coefficients, multiscale frequency banks, shifted query alignment, and non‑stationary key rotation. The authors prove that standard RoPE is time‑translation invariant and cannot capture seasonal contexts, while T‑RoPE breaks this invariance while preserving the RoPE interface. Across five public benchmarks and an industrial‑scale e‑commerce dataset, T‑RoPE outperforms baselines by up to 130 % in HR@10 and delivers significant online lift in conversion and order count.
arXiv AI
Jun 26
Rotary Position Encodings for Graphs
arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.
arXiv Machine Learning
Jul 9
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
arXiv:2607. 07678v1 Announce Type: new Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly.
arXiv Machine Learning
Jul 20
Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
arXiv:2601. 22402v2 Announce Type: replace-cross Abstract: Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation.
arXiv Computation and Language
3d ago
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...