arXiv Machine Learning By Athanasios Zeris

Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding

Read the original on arXiv Machine Learning →

arXiv:2606. 01258v1 Announce Type: new Abstract: Standard positional encodings for transformers - sinusoidal and rotary (RoPE) - treat every position as equally local: they encode where a token is, but not how far its positional influence should extend.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI
Jun 26

Rotary Position Encodings for Graphs

arXiv:2509. 22259v4 Announce Type: replace-cross Abstract: We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data.

By Isaac Reid, Arijit Sehanobish, Cederik H\"ofs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Choromanski, Richard E. Turner, Petar Veli\v{c}kovi\'c