arXiv AI
Dynamic Relational Priming Improves Transformer in Multivariate Time Series
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Oct 21
Interdomain Attention: Beyond Token-Level Key-Value Memory
The paper introduces Interdomain Attention (IA), a hybrid architecture that blends softmax attention with recurrent memory within a single layer, leveraging a kernel regression perspective to compress keys and values into a state‑space model (SSM) memory. IA reconstructs the attention matrix implicitly from memory states, preserving the computational efficiency of SSMs while achieving performance gains. Experiments on models ranging from 125 M to 1.3 B parameters show IA surpasses an S4D baseline at matched memory size, outperforms standard Transformers on perplexity and common‑sense reasoning, and matches or exceeds SSM‑based models on genomic downstream tasks.
arXiv Machine Learning
Sep 29
Progressive Memory Transformer: Memory-Aware Attention for Time-Series
The paper introduces the Progressive Memory Transformer (PMT), a transformer variant that adds writable, window‑aligned memory to expose mid‑range representations alongside token and sequence‑level outputs. PMT is trained with a hierarchical learning framework that applies separate objectives at local, mid‑range, and global scales, encouraging the model to capture fine‑grained variation, window‑level motifs, and overall sequence agreement. Experiments on seven UCR/UEA/UCI classification datasets, a cue‑retention probe, and forecasting tasks show that PMT achieves strong low‑label classification performance, competitive multi‑horizon forecasting, and evidence that its memory states encode mid‑range motifs.
arXiv AI
Jun 18
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
arXiv:2605. 11287v2 Announce Type: replace-cross Abstract: A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers.
arXiv AI
Jul 3
Self-Gating Attention for Efficient Time Series Forecasting
arXiv:2607. 02344v1 Announce Type: cross Abstract: Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps.
arXiv Machine Learning
Oct 15
Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
arXiv:2610.19603v1 Announce Type: new Abstract: Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remain...
arXiv AI
Sep 25
TimeBraid: Unifying Time Series and Language for Understanding and Forecasting
TimeBraid is a family of unified models that combine pretrained language models with pretrained time‑series foundation models using interleaved global residual attention layers. The models inherit instruction following, reasoning, and continuous‑signal perception, fusing both modalities into a shared representation space for understanding and generation. The design focuses on aligning representation spaces, grounding language in temporal structure, balancing understanding with generation, and maintaining stable joint optimization, supported by 2.2 M curated series‑text pairs and 4.9 M instruction‑tuning samples. Across diverse benchmarks, TimeBraid competes with larger general‑purpose and task‑specific models.