OpenAI Blog
Learning sparse neural networks through L₀ regularization
Read the original on OpenAI Blog →The Flow has not summarised this story yet — read it at OpenAI Blog.
Related stories
arXiv AI
Sep 24
Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon
The paper presents a near-complete, nonasymptotic generalization theory for multilayer neural networks using path regularization, applicable to broad Lipschitz loss functions without requiring bounded loss or extreme network hyperparameters. It provides an explicit upper bound that addresses approximation rates in generalized Barron spaces and demonstrates the double descent phenomenon for ReLU networks. The authors claim near-minimax optimality for regression problems and plan to establish matching lower bounds in future work.
arXiv Machine Learning
Aug 11
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
arXiv:2502. 11152v4 Announce Type: replace-cross Abstract: The optimization foundations of deep linear networks have recently received significant attention.
arXiv Machine Learning
Sep 17
Regularized Least Squares Training of Quadratic Neural Networks with Applications to System Identification
The paper introduces a least‑squares method for training quadratic neural networks with regularization, providing a lower bound on the training optimization problem when the regularization coefficient is positive. It delivers closed‑form expressions for both the approximate solution and its sensitivity to data errors, and shows that the solution is optimal when the regularization coefficient is zero. The approach offers computational advantages over iterative methods like backpropagation and is validated on a nonlinear system identification example.
arXiv Machine Learning
Aug 13
Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
arXiv:2608. 11479v1 Announce Type: new Abstract: We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset.
arXiv Machine Learning
Jun 10
Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss
arXiv:2402. 00152v5 Announce Type: replace Abstract: Constructing the architecture of a neural network is a challenging pursuit for the machine learning community, and the dilemma of whether to go deeper or wider remains a persistent question.
arXiv Machine Learning
Jun 8
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
arXiv:2505. 21423v3 Announce Type: replace Abstract: The remarkable generalization properties of overparameterized networks are often attributed to implicit biases, such as norm minimization at small learning rates and low sharpness in the Edge-of-Stability regime.