← Back to all news
arXiv Computation and Language By Zhongping Ji

HySPE: Positional Encoding via Symplectic Dual Shears

Read the original on arXiv Computation and Language →

HySPE introduces a hyperbolic symplectic positional encoding that uses dual shears to create a conformally symplectic contraction with two spectral decay rates per channel pair. It eliminates exponential drift by diagonalizing the operator and applying blockwise coordinate rebasing, ensuring length‑independent numerical bounds while matching the forward latency of RoPE. Experiments on TinyShakespeare and a 51M‑parameter Transformer show HySPE maintains stable perplexity at long sequence lengths, outperforming RoPE in extrapolation scenarios.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Jun 25

Why Do Accumulated Transformations Extrapolate?

arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.

By Mahesh Godavarti
More like this →
arXiv Machine Learning
Oct 16

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

The paper investigates the use of partial Rotary Positional Embedding (RoPE) in transformer models, showing that applying RoPE to only about 10% of hidden dimensions can save up to ten times the memory of full RoPE while achieving comparable final loss. The authors systematically analyze training dynamics and convergence across various architectures, model sizes, sequence lengths, and datasets, finding consistent performance trends. They also note that completely omitting positional encoding (NoPE) can cause loss spikes, which are mitigated by minimal RoPE usage or QK‑Norm application.

By Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander
llmsrag
More like this →