arXiv:2601. 11618v2 Announce Type: replace-cross Abstract: Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied).
By Luis Rosario Freytes
arXiv:2603. 17433v2 Announce Type: replace-cross Abstract: Transformer models have redefined sequence learning, yet dot-product self-attention introduces a quadratic token-mixing bottleneck for long-context time-series.
By Dibakar Sigdel
The paper introduces a bounded spectral framework to analyze rotary attention in transformer language models, focusing on phase alignment, hidden‑state continuity, and semantic drift. It identifies ordered hidden‑state sequences as suitable domains for spectral decomposition, derives the Rotary Position Embedding (RoPE) attention score as a sum of magnitude‑weighted cosine terms, and proves a local stability lemma linking bounded phase displacement to pre‑softmax score degradation. By defining complex modal coordinates and a weighted coherence functional, the work distinguishes representational continuity from execution‑boundary admissibility, offering a theoretical program for when spectral structure explains continuity and when external governance is required.
By Abraham Chachamovits
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
By Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh, Tan Lai Ngoc, Tan M. Nguyen
arXiv:2607. 11897v1 Announce Type: new Abstract: Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory.
By Tiantian Zhang
arXiv:2606. 17961v1 Announce Type: cross Abstract: Positional encoding is a fundamental component of Transformer architectures, as it injects information about the spatial or sequential arrangement of inputs.
By Andrea Santomauro, Luigi Portinale, Giorgio Leonardi
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
By Mahesh Godavarti
The paper proposes a principled way to design hybrid transformer architectures that combine Full Attention (FA) and Linear Attention (LA). By introducing two intervention metrics—RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD)—the authors identify a clear taxonomy of retrieval and positional heads, defining a Global Positional Band (GPBand) that aligns with training-length positional scales. Using these insights, they build a Head‑wise Hybrid Architecture (HwH) that assigns FA to global retrieval and LA to local positional modeling, achieving strong language modeling, improved retrieval, and superior zero‑shot long‑context extrapolation compared to standard Transformers and other hybrids.
By Runlin Shi, Bojian Yin, Guoqi Li
arXiv:2509.12635v4 Announce Type: replace-cross
Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention sco...
By Yu Wang, Sheng Shen, R\'emi Munos, Hongyuan Zhan, Yuandong Tian
arXiv:2606. 23957v1 Announce Type: new Abstract: Learning Koopman operators with autoencoders enables linear prediction in a latent space, but long-horizon rollouts often drift off the learned manifold, leading to phase and amplitude errors on systems with switching, continuous spectra, or strong transients.
By Mohammed Nagdi, Evangelos-Marios Nikolados, Alexey Yermakov, Mars Gao, Nathan Kutz, Filippo Menolascina
arXiv:2607. 07740v1 Announce Type: cross Abstract: Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints.
By Haozhan Tang, Zerui Wang, Yuxian Gu, Song Han, Han Cai
arXiv:2606. 18694v1 Announce Type: new Abstract: A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured departures from agreement.
By Joshua Nunley