The paper introduces Sliding Networks, a family of residual neural network structures that aim to preserve both feature learning and feature diversity while scaling width and depth. By maintaining a limiting ratio of residual width to the product of hidden width and depth, the authors derive a joint scaling recipe that converges to a stochastic differential equation (SDE) limit with locally maximal feature learning. Experiments on residual-MLP and language models demonstrate that this scaling approach allows redistribution of computational resources between width and depth, though performance varies when parameters are regrouped at a fixed count.
arXiv:2602. 05970v2 Announce Type: replace-cross Abstract: Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies.
arXiv:2510. 24616v4 Announce Type: replace-cross Abstract: For four decades statistical physics has been providing a framework to analyse neural networks.
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
arXiv:2602. 07697v3 Announce Type: replace-cross Abstract: Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights.
arXiv:2607. 16720v1 Announce Type: new Abstract: Understanding deep neural networks remains a central challenge in machine learning.
arXiv:2403.04545v4 Announce Type: replace Abstract: Scaling factors in residual branches have emerged as a prevalent method for boosting neural network performance, especially in normalization-free a...
Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, a natural question is: can DNNs attain feature-learning and prediction consistency comparable to that of classical models?
Transferable Graph Metanetworks extend the graph metanetwork paradigm to predict properties of neural networks of varying widths. By enforcing invariance to width-dependent representations and continuity across similar functions, the proposed modifications enable robust size generalization, especially for networks trained under the maximal‑update parameterization (μP) up to 42× the training width. The authors provide empirical evidence across multiple tasks and theoretical guarantees in the infinite‑width limit, explaining both successes and limitations under different parameterizations.
We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.
arXiv:2610.35514v1 Announce Type: new Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.