arXiv Statistics ML
Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
4d ago
Can Representation Learning Decouple from Loss Minimization? Polar Updates Have an Answer
arXiv:2609.36240v1 Announce Type: new Abstract: Does representation learning stop when the training loss stops improving? We study this question for matrix Muon, whose polar-normalised updates have a...
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv AI
2d ago
Removing spurious minima for planar features by skip connections
arXiv:2610.01728v1 Announce Type: cross Abstract: Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple...
arXiv AI
Aug 10
Hidden Gauge Controls Feature Specialization in ReLU Networks
arXiv:2608. 06766v1 Announce Type: cross Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units.
arXiv AI
Sep 11
Teacher Geometry Shapes Learnability in Teacher-Student Networks
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
arXiv Machine Learning
Jul 28
SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
arXiv:2607. 23346v1 Announce Type: new Abstract: Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure.