← Back to all news
arXiv Machine Learning By Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

Read the original on arXiv Machine Learning →

The paper investigates the use of partial Rotary Positional Embedding (RoPE) in transformer models, showing that applying RoPE to only about 10% of hidden dimensions can save up to ten times the memory of full RoPE while achieving comparable final loss. The authors systematically analyze training dynamics and convergence across various architectures, model sizes, sequence lengths, and datasets, finding consistent performance trends. They also note that completely omitting positional encoding (NoPE) can cause loss spikes, which are mitigated by minimal RoPE usage or QK‑Norm application.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 23

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.

By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li
llmsrag
More like this →
arXiv Computation and Language
Oct 7

WavePrune: One period is often enough for RoPE

arXiv:2610.06963v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific fr...

By Guancheng Du, Luotian Huang, Shaowen Wang, Si Li, Kaifeng Lyu
ragefficiency
More like this →
arXiv AI
Oct 12

Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Kronecker Product Attention (KPA) is introduced as a memory‑efficient method for compressing the key‑value cache in autoregressive models. By representing key and value activations as sums of Kronecker products of compact, token‑dependent matrix factors, KPA generalizes Tensor Product Attention and can capture richer block‑structured patterns. Empirical results show that KPA matches or surpasses existing techniques such as MHA, MQA, GQA, and MLA across multiple model scales and benchmarks, achieving up to 10× compression while preserving relative‑position information through a structured rotary embedding.

By Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh, Vahid Partovi Nia, Parsa Omidi
ragefficiencybenchmarks
More like this →