← Back to all news
arXiv Machine Learning By Julian D. Schiller, Matthias A. M\"uller

Truncated backpropagation through time induces a two-sided gradient turnpike property

Read the original on arXiv Machine Learning →

Truncated backpropagation through time (TBPTT) limits gradient calculations to shorter temporal windows, creating two boundary effects: missing past information disturbs forward state evolution, and truncating backward recursion omits future information. By decoupling the interval over which gradients are averaged from the surrounding context, and assuming exponential decay of state‑transition Jacobians, the authors prove a two‑sided gradient turnpike property: the error between truncated and full‑context gradients decays exponentially from both boundaries. They provide bounds on the averaged update gradient, guaranteeing gradient alignment and approximate stationarity, and under unbiased sampling and a Polyak‑Łojasiewicz condition, they bound the optimality gap, with numerical experiments on the Wiener‑Hammerstein benchmark illustrating the trade‑off between predictive performance and computational cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Sep 8

Learning Length-Extrapolatable Recurrent Models

The paper investigates why recurrent models often fail to generalize beyond their training horizon, noting that vanishing or exploding gradients are not the sole cause. It introduces the concept of state credit—the influence of future losses on earlier recurrent states—and proposes Credit Stabilization through Time (CST), a method that rescales this signal during backpropagation to stabilize its norm. Experiments on synthetic and real data show that CST enables models to perform well up to 128 times longer than their training length.

More like this →
arXiv Machine Learning
Sep 14

Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting

The paper introduces Internal Dual-Wiener routing (Internal‑DW), a backward‑only method that weight‑balances internal gradient routes in autoregressive forecasting. By estimating bounded Wiener gains for identity and nonlinear paths, it suppresses unpredictable noise while preserving predictable learning signals, reducing forecast error by 5.2%–13.8% on four weak‑drive testbeds compared to full BPTT and outperforming gradient clipping, Jacobian regularization, and truncated BPTT in most cases. The approach shows that long‑horizon supervision can be effective without trusting every backward gradient equally.

By Junhao Zhao, David Michael Simberg, Jacob Kang, Colin Connor Kurniawan, Nan Xu
benchmarks
More like this →
arXiv Machine Learning
Oct 16

Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

The paper investigates weight perturbation (WP) as a derivative-free alternative to backpropagation for training fully connected MNIST networks. In shallow architectures, large-batch WP attains about 97% test accuracy, close to backpropagation, but under a ten‑epoch budget WP lags behind (90.2% vs. 97.4%). The study also shows diminishing returns beyond a batch size of 4,000 and that learning‑rate reduction and averaging of perturbation updates can stabilize deeper networks, though they do not yet match backpropagation accuracy.

By Patrick Inoue
More like this →