arXiv:2610. 21820v1 Announce Type: new Abstract: How depth and width of neural networks shape learned representations is usually studied only after training.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
The paper introduces Sliding Networks, a family of residual neural network structures that aim to preserve both feature learning and feature diversity while scaling width and depth. By maintaining a limiting ratio of residual width to the product of hidden width and depth, the authors derive a joint scaling recipe that converges to a stochastic differential equation (SDE) limit with locally maximal feature learning. Experiments on residual-MLP and language models demonstrate that this scaling approach allows redistribution of computational resources between width and depth, though performance varies when parameters are regrouped at a fixed count.
The paper introduces a pointwise generalization theory for fully connected deep neural networks, using a pointwise Riemannian Dimension derived from eigenvalues of learned feature representations across layers. This framework provides hypothesis-dependent, representation-aware generalization bounds that are significantly tighter than traditional size- or norm-based approaches, both theoretically and experimentally. The authors analytically identify structural properties that explain deep networks’ tractability and empirically show that the pointwise Riemannian Dimension captures feature compression, over‑parameterization effects, and optimizer bias.
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv:2610.35514v1 Announce Type: new Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...
arXiv:2604. 20219v2 Announce Type: replace Abstract: Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear.