Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

The paper investigates gradient descent dynamics in a minimal two‑neuron ReLU teacher‑student model with isotropic Gaussian inputs. It identifies two key directions—an easy bisector direction carrying the main signal and a hard splitting direction governing specialization—and shows that every critical point with non‑zero student weights is either a global minimum or a saddle. The study tracks the gradient‑flow trajectory, revealing that student neurons initially converge to a bisector saddle before escaping and specializing to match the teacher neurons, thereby explaining the multi‑phase symmetry‑breaking behavior in this simple setting.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.