arXiv:2610. 08816v1 Announce Type: new Abstract: Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately?
arXiv:2602.06797v3 Announce Type: replace-cross Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...
arXiv:2610.06894v1 Announce Type: cross Abstract: A central question in the prediction of stochastic processes is the extent to which past information can improve the probability of correctly predict...
arXiv:2605. 05113v2 Announce Type: replace Abstract: We study signal propagation in linear recurrent models at finite width.
The paper shows that a memory state that reproduces every current prediction can still fail to learn from subsequent observations. It characterizes when a fixed‑dimensional continuous recurrent state exists for exact cumulative learning and attention, proving that for smooth convex residual losses the state exists only when the loss is polynomial. The work further quantifies the dimensionality required for various loss functions, demonstrates unbounded requirements for nonpolynomial losses, and analyzes specific cases such as ridge regression and kernel attention.
arXiv:2610.07623v1 Announce Type: cross Abstract: Probability forecasts are calibrated when predicted probabilities match empirical outcome frequencies: among events assigned a probability $p$, we'd...
Fast Weight Attention for Continual Learning introduces recurrent fast‑weight memories and selective state‑space models that compress expanding context into a fixed‑size recurrent state, enabling an online learning rule for state transitions. The paper derives normalized first‑order updates for squared‑error regression and negative inner‑product objectives, presenting several variants (Falcon‑1, Falcon‑2, Falcon‑3 and their inner‑product counterparts) with recurrent, masked‑parallel, and chunk‑parallel implementations. These methods demonstrate competitive performance in language modeling and improved length extrapolation on variable‑digit addition tasks.
arXiv:2606. 29519v1 Announce Type: new Abstract: Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data.
arXiv:2609.36314v1 Announce Type: new Abstract: State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for...
arXiv:2602. 16864v2 Announce Type: replace-cross Abstract: Time series (TS) modeling has come a long way from early statistical, mainly linear, approaches to the current trend in TS foundation models.
arXiv:2607. 24892v1 Announce Type: cross Abstract: Text-conditioned time-series forecasting predicts a series from both its numerical history and natural-language context, allowing forecasts to account for events and constraints that the past alone cannot reveal.
arXiv:2606. 14488v1 Announce Type: cross Abstract: Recent finite-time analyses of nonlinear two-time-scale stochastic approximation show that under contractive assumptions the slow iterate $Y_k$ with stepsizes $\beta_k=\Theta(k^{-1})$ and $\alpha_k=\Theta(k^{-a})$, $a\in(1/2,1)$, generally satisfies a mean-square rate of order $k^{-a}$; decoupled $k^{-1}$ rates require strong local linearity.
The paper investigates how much learned memory is required to leverage additional data in autoregressive prediction models. It introduces a predictive‑energy spectrum that jointly governs data and memory scaling, proving a minimax law that links the number of prediction blocks and the size of the learned state to this spectrum. The authors demonstrate that optimal bit allocation and masked query‑key attention mechanisms realize this law, and they provide experimental evidence across multiple pretrained‑model scales.