arXiv:2608. 04451v1 Announce Type: cross Abstract: We give a negative solution to MAIS-O60.
We give a negative solution to MAIS-O60. We first construct an example in which an initially active ReLU neuron becomes completely inactive in finite time and thereafter remains frozen at a limit whose Fourier energy is equally distributed among all nonzero real frequency classes.
arXiv:2610. 11475v1 Announce Type: new Abstract: Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining.
Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism.
arXiv:2609. 12996v1 Announce Type: new Abstract: We prove a full Adam theorem for spectral heavy-tail onset in a closed Gaussian Stein-Hermite teacher-student state-evolution model.
arXiv:2609. 12994v1 Announce Type: new Abstract: Heavy-tailed empirical spectral densities of neural-network weight matrices are widely used as diagnostics of implicit self-regularization, but the step complexity of heavy-tail emergence remains poorly understood.
The paper introduces a new method called uniform‑phase initialization for deep neural networks with sine activations, eliminating the need for the Central Limit Theorem and fully decoupling layers. This approach avoids distributional approximation errors and coupling between layers, leading to stable weight initialization. Experiments show that models using this initialization outperform state‑of‑the‑art methods on image and audio fitting tasks and remain competitive without tuning, while also supporting μP width scaling.
arXiv:2610. 31663v1 Announce Type: cross Abstract: Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing.
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv:2607. 04135v1 Announce Type: cross Abstract: The remarkable ability of modern neural networks to generalize improves with increasing network capacity, even when the number of model parameters or effective degrees of freedom exceeds the number of training data points.
The paper investigates looped transformers, which perform inference by repeatedly applying a weight‑tied map, making their computation a dynamical process. It shows that identical architectures trained to the same accuracy can converge to distinct dynamical phases—one governed by a saddle‑node fold and another by a Neimark‑Sacker transition—each with different compute scaling behaviors. The study derives a parameter‑free relation linking relaxation time and spectral gap in the fold phase and demonstrates how critical slowing down leads to a workload‑level tail distribution, while the Neimark‑Sacker phase eliminates this scaling law.
arXiv:2605. 10931v2 Announce Type: replace-cross Abstract: Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models.
arXiv:2610.20439v1 Announce Type: new Abstract: We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well:...