arXiv Machine Learning
Estimating Implicit Regularization in Deep Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 8
Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization
arXiv:2505. 21423v3 Announce Type: replace Abstract: The remarkable generalization properties of overparameterized networks are often attributed to implicit biases, such as norm minimization at small learning rates and low sharpness in the Edge-of-Stability regime.
arXiv Machine Learning
Jul 21
Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks
arXiv:2607. 16720v1 Announce Type: new Abstract: Understanding deep neural networks remains a central challenge in machine learning.
arXiv AI
Sep 24
Path Regularization: A Near-Complete and Optimal Nonasymptotic Generalization Theory for Multilayer Neural Networks and Double Descent Phenomenon
The paper presents a near-complete, nonasymptotic generalization theory for multilayer neural networks using path regularization, applicable to broad Lipschitz loss functions without requiring bounded loss or extreme network hyperparameters. It provides an explicit upper bound that addresses approximation rates in generalized Barron spaces and demonstrates the double descent phenomenon for ReLU networks. The authors claim near-minimax optimality for regression problems and plan to establish matching lower bounds in future work.
arXiv Machine Learning
1d ago
Continual Classification Beyond the Kernel Regime: Harnessing the Sparsity Bias of L2 Regularization in Diagonal Linear Networks
arXiv:2611.06759v1 Announce Type: new Abstract: We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has lar...
arXiv Machine Learning
Jun 18
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
arXiv:2603. 04895v2 Announce Type: replace-cross Abstract: Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima.
arXiv Machine Learning
Oct 9
Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks
The paper investigates why saliency maps of adversarially‑trained neural networks appear sparse. It provides a theoretical explanation for two‑layer ReLU networks, showing that as data size and neuron count grow, minimizers converge to a Bayes classifier with minimal gradient and Barron norm. The sparsity arises because α−−‑training with λ−1 attacks yields anisotropic gradient norms that favor axis‑aligned, sparse gradients, a claim supported by experimental evaluation of gradient α‑norms and sparsity thresholds.