Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective
Read the original on arXiv Machine Learning →The paper analyzes gradient flow in a two‑layer linear neural network as the input, hidden, and output dimensions grow proportionally. Using a closed‑form solution combined with random matrix theory, it derives exact asymptotic spectral distributions for the weight matrices at all times and tracks how the learned subspace evolves. In the limit of small initialization, the study reveals a silent alignment phase where singular directions align quickly while singular values and loss stay unchanged, followed by a visible learning phase on a logarithmic timescale where singular values increase and loss decreases through a saddle‑to‑saddle dynamic that becomes continuous in the high‑dimensional limit.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.