OpenAI Blog
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Read the original on OpenAI Blog →The Flow has not summarised this story yet — read it at OpenAI Blog.
Related stories
arXiv Machine Learning
Jun 4
Path-conditioned training: a principled way to rescale ReLU neural networks
arXiv:2602. 19799v2 Announce Type: replace-cross Abstract: Despite recent algorithmic advances, we still lack principled ways to leverage the well-documented rescaling symmetries in ReLU neural network parameters.
arXiv AI
Jul 14
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.
arXiv Machine Learning
Jul 24
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
arXiv Machine Learning
Jul 10
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
arXiv:2607. 08406v1 Announce Type: new Abstract: Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients.
arXiv Machine Learning
Jun 16
On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
arXiv:2510. 01175v2 Announce Type: replace Abstract: While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited.
Hugging Face Trending Papers
Jul 9
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence.