← Back to all news
arXiv Machine Learning By Oleksandr Kholodniak, Oleksandr Prokhorov

Gate Initialisation versus Recurrent Dropout and Zoneout in Learning Long Dependencies: A Controlled Study of LSTM and GRU Networks on Synthetic Tasks

Read the original on arXiv Machine Learning →

The study investigates how gate initialisation and recurrent regularisers (dropout and zoneout) affect learning long dependencies in one‑layer LSTM and GRU networks on synthetic tasks. By running 720 factorial experiments that cross conventional and chrono gate initialisation with up to six regulariser placements, the authors find that chrono initialisation consistently enables networks to reach validation criteria within 5000 updates, whereas conventional initialisation often fails or learns slowly. Dropout generally delays learning, zoneout only helps in limited conditions, and early input sensitivity provides limited predictive power for successful training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 5

The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisition

The paper investigates how different optimizers acquire distinct generalizing relations in single‑hidden‑layer ReLU networks. By manipulating nuisance‑connected weights and performing a strict‑parity sweep, the authors show that training conditions reshape both shared failures and relative optimizer advantages, with specific interventions improving accuracy on hard stages. The study also compares optimizer behavior across tasks and an external architecture (CIFAR‑10).

By Gongyue Zhang, Honghai Liu
More like this →
arXiv Machine Learning
Oct 8

When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

The study investigates whether a neural network’s training history can better predict its future learning than its current state. Using small multilayer perceptrons trained under 42 different history regimes, the authors measured future learning at four checkpoints via a short probe that replicated the network for 100 updates on a new task. Results show that a low‑dimensional history state does not outperform a calibrated model of the current state for short‑term prediction, but in a companion screen of 1,560 synthetic regression runs, history models forecast final error better than current validation error after 12 epochs and remain comparable until 48 epochs.

By Martin Hofmann, Patrick M\"ader
More like this →