Towards Data Science
Aug 8
Before Q, K, and V: Reconstructing the Transformer
Many Transformer explainers start with the finished architecture. We ask why it looks the way it does.
Why Transformers Need Positional Encoding For Time Series: A Visual Guide
The article explains how transformers, which rely on self‑attention, can lose the natural order of time‑series data when fed scalar observations. It discusses the role of positional encoding in re‑introducing sequence order and provides a visual guide to illustrate this concept.
Related stories
arXiv AI
Jun 17
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
arXiv:2606. 17830v1 Announce Type: cross Abstract: Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence.
arXiv AI
Jun 6
Where does Absolute Position come from in decoder-only Transformers?
arXiv:2606. 06160v1 Announce Type: new Abstract: RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product.
arXiv Machine Learning
5d ago
Pattern Formation in Transformers
arXiv:2609.37921v1 Announce Type: new Abstract: What are the inductive biases of a Transformer architecture? Existing theory on how the forward pass shapes representations either considers whether Tr...
Hugging Face Blog
Jun 16, 2023
Yes, Transformers are Effective for Time Series Forecasting (+ Autoformer)
Towards Data Science
Jul 2
Time-Series LLMs, Explained with t0-alpha
t0-alpha is a decoder-style patch transformer for probabilistic time-series forecasting. Raw series are split into 32-step patches, embedded, processed through causal time-attention and group-attention layers, and decoded into future quantiles rather than a single point forecast.
arXiv Machine Learning
Sep 1
Universal Redundancies in Time Series Foundation Models
arXiv:2602.01605v2 Announce Type: replace Abstract: Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need fo...
arXiv Computation and Language
Sep 11
Distance generalization in transformers: why bother with positional encoding?
The paper investigates distance generalization in transformer models, focusing on how well they can handle changes in inter-token distances between training and inference while keeping context length fixed. Using two synthetic delay-copy tasks that require copying tokens after finite delays, the authors evaluate the impact of positional encoding schemes (RoPE, ALiBi, and NoPE), the diversity of distances seen during training, and the conditions under which distance transfer learning is beneficial or detrimental. Their comprehensive study highlights the importance of understanding the underlying mechanisms that govern distance generalization in transformers.
Hugging Face Blog
Dec 1, 2022
Probabilistic Time Series Forecasting with 🤗 Transformers
arXiv AI
Jun 9
Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
arXiv:2509. 10534v3 Announce Type: replace-cross Abstract: The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where.
arXiv Machine Learning
Aug 6
A Mechanistic Analysis of Transformers for Dynamical Systems
arXiv:2512. 21113v2 Announce Type: replace Abstract: Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective.
arXiv Machine Learning
Jun 2
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
arXiv:2606. 01532v1 Announce Type: new Abstract: Positional encoding (PE) is widely viewed as necessary for transformers to process ordered sequences: without them, the next-token map appears permutation-invariant in its context tokens.