arXiv AI
Removing spurious minima for planar features by skip connections
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 11
Teacher Geometry Shapes Learnability in Teacher-Student Networks
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
arXiv Machine Learning
Jun 30
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
arXiv Statistics ML
3d ago
Awakening of the Buddha: Subspace Learning During Population-Loss Plateaus
arXiv:2609.39408v1 Announce Type: cross Abstract: Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation...
arXiv Machine Learning
5d ago
Population loss in shallow ReLU networks: Bias & families of critical points
The paper presents a formula for the population loss in shallow ReLU networks with bias within the student‑teacher kernel model, extending earlier results by Choo and Saul (2009) and Brutzkus and Globerson (2017). It utilizes Owen’s T‑function, providing the necessary theory and a high‑precision implementation via MPFR. The study shows that adding bias strictly decreases loss, extends known families of spurious minima to biased networks, and indicates that the resulting change in landscape geometry is relatively mild, focusing on cases where the number of inputs equals the number of neurons.
arXiv Machine Learning
Jul 27
Statistical mechanics of extensive-width Bayesian neural networks near interpolation
arXiv:2505. 24849v2 Announce Type: replace-cross Abstract: For three decades statistical mechanics has been providing a framework to analyse neural networks.
arXiv Machine Learning
Jun 30
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
arXiv:2606. 30444v1 Announce Type: cross Abstract: Neural networks are known to be susceptible to over-reliance on spurious correlations.