arXiv Machine Learning
Jun 2
Multigrade Neural Network Approximation
arXiv:2601. 16884v3 Announce Type: replace Abstract: We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks.
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
arXiv:2606. 15669v1 Announce Type: cross Abstract: Modern deep neural networks rely on Euclidean scalar activations (e.
Related stories
arXiv AI
Jul 1
Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization
arXiv:2606. 32000v1 Announce Type: cross Abstract: Why do neural networks memorize algorithmic training data long before they generalize?
arXiv AI
Jul 21
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
arXiv:2607. 17822v1 Announce Type: cross Abstract: While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion.
arXiv Machine Learning
Jul 2
Zeroth-Order Optimization at the Edge of Stability
arXiv:2604. 14669v2 Announce Type: replace Abstract: Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored.
arXiv Machine Learning
Jun 24
Layer-wise Geometric Approximation Rates for Deep Networks
arXiv:2604. 20219v2 Announce Type: replace Abstract: Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear.
arXiv Machine Learning
Jul 24
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
arXiv Machine Learning
Jul 21
Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks
arXiv:2607. 16720v1 Announce Type: new Abstract: Understanding deep neural networks remains a central challenge in machine learning.
arXiv Machine Learning
Jul 10
Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
arXiv:2607. 07845v1 Announce Type: new Abstract: The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive.
arXiv Machine Learning
Jun 25
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
arXiv Machine Learning
Jun 3
Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks
arXiv:2602. 10949v2 Announce Type: replace-cross Abstract: Effective initialization in deep networks requires an understanding of random neural networks.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
Hugging Face Trending Papers
Jun 8
Preserving Plasticity in Continual Learning via Dynamical Isometry
Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning.