arXiv:2610.35514v1 Announce Type: new
Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...
By Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
arXiv:2601.06961v2 Announce Type: replace
Abstract: The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization o...
By Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
The paper investigates how nonlinearity in attention mechanisms affects scaling laws in large language models. It finds that nonlinear attention leads to an inverse-depth decay of loss across various data spectra, enabling models to learn both strong and weak spectral directions simultaneously. This behavior contrasts with linear-attention models, which rely on a power-law data spectrum and learn directions sequentially.
By Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore
arXiv:2609.37745v1 Announce Type: new
Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent...
By Hyunseok Lee, Mihir Basil, Yizhou Liu, Jeff Gore
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
By Liu Ziyin, Yizhou Xu, Tomaso Poggio, Isaac Chuang
arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.
By Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore