arXiv Statistics ML
Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Jul 10
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.
arXiv AI
Oct 9
Emergent Inverse-Depth Scaling From Nonlinearity In Attention
The paper investigates how nonlinearity in attention mechanisms affects scaling laws in large language models. It finds that nonlinear attention leads to an inverse-depth decay of loss across various data spectra, enabling models to learn both strong and weak spectral directions simultaneously. This behavior contrasts with linear-attention models, which rely on a power-law data spectrum and learn directions sequentially.
arXiv Machine Learning
Jun 5
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv AI
Jun 4
An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization
arXiv:2606. 04409v1 Announce Type: cross Abstract: Modern deep neural networks usually have large parameter scales and nonlinear hierarchical structures, and they have achieved strong performance in computer vision.
arXiv AI
Aug 11
On the Infinite Width and Depth Limits of Predictive Coding Networks
arXiv:2602. 07697v3 Announce Type: replace-cross Abstract: Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights.
arXiv Machine Learning
Jun 2
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.