arXiv Machine Learning
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 10
A law of robustness for two-layer neural networks with arbitrary weights
arXiv:2607. 07778v1 Announce Type: new Abstract: Bubeck, Li and Nagaraj conjectured that, for generic data, any two-layer neural network with $m$ neurons that fits $n$ noisy labels must have Lipschitz constant at least of order $\sqrt{n/m}$, with no restriction on the size of the weights.
Hugging Face Trending Papers
Jul 7
A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel
A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its $\textbf{Fourier complexity}$, which controls NTK kernel regression, and its $\textbf{architectural complexity}$, which controls learning over depth-$L$, width-$w$ ReLU networks with the variation norm of the weights bounded by $R$.
arXiv Machine Learning
Jul 14
Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
arXiv:2607. 10589v1 Announce Type: cross Abstract: In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter $N$ and the depth parameter $L$, thereby granting greater architectural flexibility.
arXiv Machine Learning
Jul 8
A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel
arXiv:2607. 06382v1 Announce Type: cross Abstract: A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking.
arXiv Machine Learning
Aug 14
From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks
arXiv:2602. 17596v2 Announce Type: replace Abstract: We study pathwise connectivity of sublevel sets for one-hidden-layer ReLU networks with constrained first-layer weights and an $\ell_1$ penalty on the output layer.
arXiv Machine Learning
Jul 21
The Value of Depth in Message Passing on Sparse Graphs: A Kesten-Stigum Dichotomy
arXiv:2607. 16676v1 Announce Type: cross Abstract: How deep does a graph neural network need to be on a sparse graph?