arXiv Machine Learning
Jun 24
Selective Rotary Position Embedding
arXiv:2511. 17388v3 Announce Type: replace-cross Abstract: Position information is essential for language modeling.
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
arXiv:2607. 10134v1 Announce Type: new Abstract: Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models.
Related stories
arXiv Machine Learning
Aug 4
RAP: KV-Cache Compression via RoPE-Aligned Pruning
arXiv:2602. 02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache.
arXiv AI
Jun 9
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
arXiv:2509. 10534v3 Announce Type: replace-cross Abstract: The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where.
arXiv Machine Learning
Jun 25
Why Do Accumulated Transformations Extrapolate?
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
arXiv AI
Jul 24
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
arXiv AI
Jul 23
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv AI
Jun 26
Rotary Position Encodings for Graphs
arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.
arXiv Machine Learning
Jul 9
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
arXiv:2607. 07678v1 Announce Type: new Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly.
arXiv Machine Learning
Jul 20
Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
arXiv:2601. 22402v2 Announce Type: replace-cross Abstract: Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation.
Hugging Face Trending Papers
Aug 3
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position. Aligned with this structure, we propose the per-RoPE-wavelength distance window: it prunes the query--key inner-product terms beyond a wavelength-proportional distance.
arXiv Machine Learning
Aug 5
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
arXiv:2608. 02947v1 Announce Type: new Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position.
arXiv AI
Jul 7
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.