arXiv Computation and Language
WavePrune: One period is often enough for RoPE
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Aug 4
RAP: KV-Cache Compression via RoPE-Aligned Pruning
arXiv:2602. 02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache.
arXiv Machine Learning
Aug 5
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
arXiv:2608. 02947v1 Announce Type: new Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position.
Hugging Face Trending Papers
Aug 3
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position. Aligned with this structure, we propose the per-RoPE-wavelength distance window: it prunes the query--key inner-product terms beyond a wavelength-proportional distance.
arXiv AI
Jul 23
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv AI
Jun 9
From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
arXiv:2606. 09508v1 Announce Type: new Abstract: Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts.
arXiv Machine Learning
4d ago
Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
The paper investigates the use of partial Rotary Positional Embedding (RoPE) in transformer models, showing that applying RoPE to only about 10% of hidden dimensions can save up to ten times the memory of full RoPE while achieving comparable final loss. The authors systematically analyze training dynamics and convergence across various architectures, model sizes, sequence lengths, and datasets, finding consistent performance trends. They also note that completely omitting positional encoding (NoPE) can cause loss spikes, which are mitigated by minimal RoPE usage or QK‑Norm application.