Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking
Read the original on arXiv Machine Learning →The paper investigates why Hessian spectra of trained deep learning models consistently show a bulk of eigenvalues near zero and a few isolated outliers. It proposes that these patterns arise from a hidden, highly symmetric reference configuration; when the model is perturbed (e.g., by architecture, data, or metric changes), the symmetry is broken, producing the observed cluster hierarchy. The authors analyze this mechanism in three‑layer ReLU networks and extend it to convolutional, graph, transformer models, the NTK, and related matrices such as layerwise Hessians and the Gauss‑Newton matrix.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.