← Back to all news
arXiv Machine Learning By Michael Field

Population loss in shallow ReLU networks: Bias & families of critical points

Read the original on arXiv Machine Learning →

The paper presents a formula for the population loss in shallow ReLU networks with bias within the student‑teacher kernel model, extending earlier results by Choo and Saul (2009) and Brutzkus and Globerson (2017). It utilizes Owen’s T‑function, providing the necessary theory and a high‑precision implementation via MPFR. The study shows that adding bias strictly decreases loss, extends known families of spurious minima to biased networks, and indicates that the resulting change in landscape geometry is relatively mild, focusing on cases where the number of inputs equals the number of neurons.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 11

Teacher Geometry Shapes Learnability in Teacher-Student Networks

The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."

By Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea
More like this →