The paper investigates why Hessian spectra of trained deep learning models consistently show a bulk of eigenvalues near zero and a few isolated outliers. It proposes that these patterns arise from a hidden, highly symmetric reference configuration; when the model is perturbed (e.g., by architecture, data, or metric changes), the symmetry is broken, producing the observed cluster hierarchy. The authors analyze this mechanism in three‑layer ReLU networks and extend it to convolutional, graph, transformer models, the NTK, and related matrices such as layerwise Hessians and the Gauss‑Newton matrix.
arXiv:2607. 07845v1 Announce Type: new Abstract: The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive.
arXiv:2611.07410v1 Announce Type: new Abstract: Symmetry governs the organization of physical structures, while symmetry breaking reveals their transformations. In geometric deep learning, symmetry i...
Symmetry governs the organization of physical structures, while symmetry breaking reveals their transformations. In geometric deep learning, symmetry imposes rank constraints on equivariant representa...
The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings.
arXiv:2506. 13139v3 Announce Type: replace-cross Abstract: Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data and rely on overparameterized models, where classical low-dimensional intuitions break down.
arXiv:2607. 13631v1 Announce Type: new Abstract: The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc.
arXiv:2606. 30226v1 Announce Type: new Abstract: Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics.
The paper investigates how machine learning models can regress scale‑free processes, such as earthquakes or avalanches, focusing on predicting rare, large events that require extrapolation. It studies two self‑similar systems: a 2‑dimensional fractional Gaussian field and the Abelian sandpile model. Experiments compare existing architectures (U‑net, Riesz network) with new proposals (wavelet‑based Graph Neural Network, Fourier embedding, Fourier‑Mellin Neural Operator) to identify spectral bias and coarse‑graining challenges and suggest inductive biases to address them.
The paper presents a theoretical analysis of symmetric autoencoders, a class of deep learning architectures frequently used in machine learning tasks. It distinguishes between different symmetric designs and shows that the reconstruction error of orthonormal symmetric autoencoders can be interpreted via the Eckart‑Young‑Schmidt theorem. Building on this insight, the authors propose an EYS‑based initialization strategy using repeated SVD, and validate its effectiveness through numerical experiments comparing it to conventional deep autoencoders.
arXiv:2605. 01288v3 Announce Type: replace Abstract: In deep networks with small initialization, training exhibits long plateaus separated by sharp feature-acquisition transitions.
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv:2410. 06665v4 Announce Type: replace-cross Abstract: This paper explores the characterization of equivariant linear layers for representations of permutations and related groups.