The paper investigates why recurrent models often fail to generalize beyond their training horizon, noting that vanishing or exploding gradients are not the sole cause. It introduces the concept of state credit—the influence of future losses on earlier recurrent states—and proposes Credit Stabilization through Time (CST), a method that rescales this signal during backpropagation to stabilize its norm. Experiments on synthetic and real data show that CST enables models to perform well up to 128 times longer than their training length.
arXiv:2609.09157v1 Announce Type: new
Abstract: Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their t...
By Hanwen Jiang
The paper introduces Internal Dual-Wiener routing (Internal‑DW), a backward‑only method that weight‑balances internal gradient routes in autoregressive forecasting. By estimating bounded Wiener gains for identity and nonlinear paths, it suppresses unpredictable noise while preserving predictable learning signals, reducing forecast error by 5.2%–13.8% on four weak‑drive testbeds compared to full BPTT and outperforming gradient clipping, Jacobian regularization, and truncated BPTT in most cases. The approach shows that long‑horizon supervision can be effective without trusting every backward gradient equally.
By Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
arXiv:2602. 18131v2 Announce Type: replace Abstract: Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware.
By Tom Potter, Oliver Rhodes
arXiv:2608. 07420v1 Announce Type: new Abstract: World models are expected to support imagination over extended temporal horizons, yet most are still trained through local few-step prediction objectives and deployed by recursively rolling out their own predictions.
By Xinyi Li, Zaishuo Xia, Chenjie Hao, Yubei Chen
The paper investigates weight perturbation (WP) as a derivative-free alternative to backpropagation for training fully connected MNIST networks. In shallow architectures, large-batch WP attains about 97% test accuracy, close to backpropagation, but under a ten‑epoch budget WP lags behind (90.2% vs. 97.4%). The study also shows diminishing returns beyond a batch size of 4,000 and that learning‑rate reduction and averaging of perturbation updates can stabilize deeper networks, though they do not yet match backpropagation accuracy.
By Patrick Inoue
The paper argues that the policy governing how tensors are rounded and reused during the backward pass—termed the backward‑state policy—is an integral part of the learning algorithm, not merely a memory or precision detail. Experiments with 390 M‑parameter models show that whether the backward pass reuses a forward’s rounded output or generates a new rounding can decisively affect training success, even when copy accuracy is high. The authors propose a method to determine, for each operator, which value should be read or substituted to preserve the correct gradient, and validate these predictions on PyTorch and Transformer Engine.
By Shuxiao Xie, Shuyang Xie, Dezhi Ran, Wei Yang, Tao Xie
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.
By Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan
arXiv:2606. 08934v1 Announce Type: new Abstract: Recurrent neural networks maintain a hidden state $h_t$, but its probabilistic meaning is often unclear.
By Yuan-chin Ivan Chang
While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored. Existing analyses often rely on double-loop architectures and invoke a linear entropy penalty.
arXiv:2608. 19762v1 Announce Type: cross Abstract: A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states.
By Kang Liu, Suyan Li
arXiv:2607. 16612v1 Announce Type: cross Abstract: Backpropagation makes training deep networks memory intensive because it must store intermediate activations.
By Tian Qin, Wei-Min Huang