arXiv:2608. 15239v1 Announce Type: new Abstract: Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex.
arXiv:2610.35514v1 Announce Type: new Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...
arXiv:2601.06961v2 Announce Type: replace Abstract: The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization o...
The paper investigates how nonlinearity in attention mechanisms affects scaling laws in large language models. It finds that nonlinear attention leads to an inverse-depth decay of loss across various data spectra, enabling models to learn both strong and weak spectral directions simultaneously. This behavior contrasts with linear-attention models, which rely on a power-law data spectrum and learn directions sequentially.
arXiv:2609.37745v1 Announce Type: new Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent...
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.
arXiv:2606. 29519v1 Announce Type: new Abstract: Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data.
arXiv:2607. 08843v1 Announce Type: new Abstract: In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space.
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.
arXiv:2605.22341v2 Announce Type: replace Abstract: Hard-label classification is usually trained with smooth surrogate losses such as softmax cross-entropy. We isolate a mechanism by which this misma...
How neural representations preserve the structure of input changes connects representation analysis with internal intervention. We study operable representational content through compatible actions of...
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.