arXiv:2606. 05219v1 Announce Type: new Abstract: Recent analyses of multi-pathway Deep Linear Networks use Gradient Flow to predict a "winner-takes-all" specialization in which path symmetry breaks and each feature concentrates in a single pathway.
arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.
arXiv:2606. 04754v1 Announce Type: new Abstract: Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged.
arXiv:2606. 19941v1 Announce Type: new Abstract: Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations.
arXiv:2607. 04993v1 Announce Type: cross Abstract: Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them.
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.
arXiv:2501. 07400v2 Announce Type: replace-cross Abstract: We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations.
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
arXiv:2607. 03613v1 Announce Type: new Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression.
The paper models the dynamics of Stochastic Gradient Descent (SGD) as a percolation process, showing that architectural symmetries cause subnetworks to merge in discrete blocks rather than sequentially. These structural transitions produce variance spikes in a macroscopic order parameter, analogous to physical phase transitions. The authors also demonstrate that this trapping mechanism and its scaling cascade apply to Adam and AdamW under a heavy‑tailed noise model.
arXiv:2606. 27759v1 Announce Type: new Abstract: Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen.