arXiv Statistics ML
Width and Depth Limits Commute During Training in Linear Residual Networks
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Oct 23
Width and Depth Limits Commute During Training
arXiv:2610. 30729v1 Announce Type: new Abstract: Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$.
arXiv Machine Learning
Oct 23
The Mathematical Theory of Double Descent as Shrinkage, not Overfitting: Matching Bounds for Path-Regularized Multilayer Neural Networks
The paper presents a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, covering both underparameterized and overparameterized regimes without assuming bounded loss, sparsity, or width/depth limits. It explicitly models approximation error, resolves an open problem on approximation rates in generalized Barron spaces, and establishes matching upper and lower bounds that show the error curve is non‑monotone—descending, rising, then descending again—as width increases. The analysis reveals that the rise is due to shrinkage imposed by regularization rather than overfitting, and this effect cannot be eliminated by width‑independent regularization.
arXiv Machine Learning
Jun 24
Layer-wise Geometric Approximation Rates for Deep Networks
arXiv:2604. 20219v2 Announce Type: replace Abstract: Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear.
arXiv Statistics ML
5d ago
Early learning in large residual networks
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
arXiv Machine Learning
Jul 7
Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
arXiv:2607. 04597v1 Announce Type: new Abstract: In this paper, we study the universal approximation property of residual neural networks, and obtain some new results.
arXiv Machine Learning
Aug 18
Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion
arXiv:2608. 14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough?