arXiv Machine Learning
Finite-width shift of the edge of chaos in deep neural networks
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 14
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
arXiv Machine Learning
Sep 29
How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences
arXiv:2605. 05113v2 Announce Type: replace Abstract: We study signal propagation in linear recurrent models at finite width.
arXiv Machine Learning
Oct 23
Phase Transition from Random Features to Fully Trainable Networks
arXiv:2610. 31663v1 Announce Type: cross Abstract: Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing.
arXiv Machine Learning
Aug 18
Randomly initialized autoencoders: fixed points and edge-of-chaos
arXiv:2608. 14638v1 Announce Type: new Abstract: In this paper we study autoencoders, a special class of deep neural nets (DNNs) whose performance can be characterized via their fixed points.
arXiv Machine Learning
Oct 16
Attention over noisy examples: generalization and memorization in the exponential Hopfield network
arXiv:2610. 22191v1 Announce Type: cross Abstract: In Hebbian learning from examples a Hopfield network of $N$ neurons is not supplied with the $K$ archetypes it has to retrieve but with $M$ blurred examples of each, of quality $r$, and whether it can infer them depends on the load $\alpha=K/N$ (interference among them) and on the dataset entropy $\rho=(1-r^2)/(Mr^2)$ (how little the examples say about them).
arXiv Machine Learning
Jul 7
The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning
arXiv:2607. 03347v1 Announce Type: new Abstract: We consider the Multiscale Single-Index Model (MSIM), first introduced in \cite{oymak2021learning}, as a stylized model for hierarchical learning with \emph{scale separation}.