HySPE introduces a hyperbolic symplectic positional encoding that uses dual shears to create a conformally symplectic contraction with two spectral decay rates per channel pair. It eliminates exponential drift by diagonalizing the operator and applying blockwise coordinate rebasing, ensuring length‑independent numerical bounds while matching the forward latency of RoPE. Experiments on TinyShakespeare and a 51M‑parameter Transformer show HySPE maintains stable perplexity at long sequence lengths, outperforming RoPE in extrapolation scenarios.
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
The paper investigates the use of partial Rotary Positional Embedding (RoPE) in transformer models, showing that applying RoPE to only about 10% of hidden dimensions can save up to ten times the memory of full RoPE while achieving comparable final loss. The authors systematically analyze training dynamics and convergence across various architectures, model sizes, sequence lengths, and datasets, finding consistent performance trends. They also note that completely omitting positional encoding (NoPE) can cause loss spikes, which are mitigated by minimal RoPE usage or QK‑Norm application.
arXiv:2609.10305v1 Announce Type: new Abstract: Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transfor...
arXiv:2607. 10134v1 Announce Type: new Abstract: Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models.
arXiv:2609.01129v1 Announce Type: new Abstract: We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under compos...
We introduce Ananke, a representation-learning framework that scaffolds latent representations onto a structured product-torus prior, and its flagship visual backbone realization, Contractive Torus At...
arXiv:2609.24737v1 Announce Type: new Abstract: We introduce Ananke, a representation-learning framework that scaffolds latent representations onto a structured product-torus prior, and its flagship...
arXiv:2610.06963v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific fr...
arXiv:2607. 06621v1 Announce Type: new Abstract: The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$.
arXiv:2608. 11909v1 Announce Type: new Abstract: Two accounts recur in explanations of the success of rotary position embeddings (RoPE).
arXiv:2606. 24650v1 Announce Type: cross Abstract: We present Harmonic, a hierarchical state space model (SSM) for language modeling.
arXiv:2602. 02599v4 Announce Type: replace Abstract: Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache.