arXiv:2610. 30729v1 Announce Type: new Abstract: Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$.
arXiv:2610.30729v2 Announce Type: replace-cross Abstract: Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branch...
arXiv:2604. 20219v2 Announce Type: replace Abstract: Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear.
The paper presents a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, covering both underparameterized and overparameterized regimes without assuming bounded loss, sparsity, or width/depth limits. It explicitly models approximation error, resolves an open problem on approximation rates in generalized Barron spaces, and establishes matching upper and lower bounds that show the error curve is non‑monotone—descending, rising, then descending again—as width increases. The analysis reveals that the rise is due to shrinkage imposed by regularization rather than overfitting, and this effect cannot be eliminated by width‑independent regularization.
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
arXiv:2607. 04597v1 Announce Type: new Abstract: In this paper, we study the universal approximation property of residual neural networks, and obtain some new results.
arXiv:2602. 07697v3 Announce Type: replace-cross Abstract: Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights.
Transferable Graph Metanetworks extend the graph metanetwork paradigm to predict properties of neural networks of varying widths. By enforcing invariance to width-dependent representations and continuity across similar functions, the proposed modifications enable robust size generalization, especially for networks trained under the maximal‑update parameterization (μP) up to 42× the training width. The authors provide empirical evidence across multiple tasks and theoretical guarantees in the infinite‑width limit, explaining both successes and limitations under different parameterizations.
arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.
arXiv:2606. 06772v2 Announce Type: replace-cross Abstract: Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning.
arXiv:2608. 14664v1 Announce Type: new Abstract: How can we determine whether a trained neural network is already deep enough?
arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.
The paper introduces Sliding Networks, a family of residual neural network structures that aim to preserve both feature learning and feature diversity while scaling width and depth. By maintaining a limiting ratio of residual width to the product of hidden width and depth, the authors derive a joint scaling recipe that converges to a stochastic differential equation (SDE) limit with locally maximal feature learning. Experiments on residual-MLP and language models demonstrate that this scaling approach allows redistribution of computational resources between width and depth, though performance varies when parameters are regrouped at a fixed count.