arXiv Machine Learning
Jun 2
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
arXiv:2605. 29548v2 Announce Type: replace Abstract: Larger models learn tasks smaller models do not.
Neuron Populations Exhibit Divergent Selectivity with Scale
arXiv:2606. 03990v1 Announce Type: new Abstract: We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss.
Related stories
arXiv AI
Jun 16
OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens
arXiv:2604. 18827v2 Announce Type: replace-cross Abstract: Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision.
arXiv Machine Learning
Jun 25
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.
OpenAI Blog
Jan 23, 2020
Scaling laws for neural language models
arXiv Machine Learning
Jun 30
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models
arXiv:2606. 29196v1 Announce Type: new Abstract: Do language models know when they are being tested?
arXiv Machine Learning
Jun 5
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv Machine Learning
Jun 8
Sparsely gated tiny linear experts
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
arXiv AI
Jul 7
Deriving Neural Scaling Laws from the statistics of natural language
arXiv:2602. 07488v3 Announce Type: replace-cross Abstract: Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset.
arXiv AI
Jun 4
An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization
arXiv:2606. 04409v1 Announce Type: cross Abstract: Modern deep neural networks usually have large parameter scales and nonlinear hierarchical structures, and they have achieved strong performance in computer vision.
arXiv Machine Learning
Jun 9
Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics
arXiv:2602. 15253v2 Announce Type: replace Abstract: Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored.
Hugging Face Trending Papers
Jun 23
Can Scale Save Us From Plasticity Loss in Large Language Models?
The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains.
arXiv Machine Learning
Aug 7
Cross-Architecture Steering Transfer in Language Models: A Systematic Empirical Study
arXiv:2608. 05164v1 Announce Type: cross Abstract: Independently trained large language models may develop shared internal representations of semantic concepts despite architectural differences -- but whether this geometric similarity has functional consequences for cross-model behavioural control remains untested.