arXiv:2609.07755v1 Announce Type: new Abstract: Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training d...
arXiv:2504.16450v4 Announce Type: replace Abstract: We derive a differential equation that governs the evolution of the generalization gap when a model is trained by gradient descent-based methods. T...
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
arXiv:2601. 19791v4 Announce Type: replace Abstract: We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting.
arXiv:2511. 01938v3 Announce Type: replace-cross Abstract: Grokking is a puzzling phenomenon in neural networks where full generalization occurs only after a substantial delay following the complete memorization of the training data.
arXiv:2606. 05335v1 Announce Type: new Abstract: Theoretical studies of machine learning models commonly consider different limiting regimes in which the learning dynamics of gradient descent becomes theoretically tractable.
The paper introduces Neural Expansion, a geometric framework that explains how deep neural networks generalize by expanding directionally-selective ray coverage in input space. It defines Generalization Intervals (GIs) to measure stable prediction regions around training samples and shows through experiments that training progressively enlarges this coverage, especially along directions with low Jacobian singular values. This expansion accounts for correct test predictions, adversarial robustness, and handling of mislabeled or out-of-distribution data, offering a mechanistic complement to existing global theories.
The paper introduces a pointwise generalization theory for fully connected deep neural networks, using a pointwise Riemannian Dimension derived from eigenvalues of learned feature representations across layers. This framework provides hypothesis-dependent, representation-aware generalization bounds that are significantly tighter than traditional size- or norm-based approaches, both theoretically and experimentally. The authors analytically identify structural properties that explain deep networks’ tractability and empirically show that the pointwise Riemannian Dimension captures feature compression, over‑parameterization effects, and optimizer bias.
arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.
The paper investigates whether flatter loss landscapes drive better generalization in the context of Grokking. It shows that sharpness‑aware minimization (SAM) alone does not reliably induce the transition to generalization, but when combined with weight decay, SAM can accelerate this transition up to four times faster. The authors provide a theoretical analysis using a minimal two‑layer ReLU model, demonstrating that flatness alone cannot distinguish memorizing from generalizing solutions, while weight decay favors generalization and a local stability window explains SAM’s effect.
arXiv:2608.25813v3 Announce Type: replace Abstract: For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Gro...
The paper investigates how memorizing long‑tail data can aid generalization when combined with a model’s ability to compose features. It demonstrates theoretically that, in a linear setting, memorization plus composition enables correct predictions on rare test examples that involve unseen combinations of long‑tail features. Experiments with neural networks on simple datasets confirm that this effect extends beyond linear models and that a model’s compositional capability depends on its architecture.