arXiv AI
Jun 16
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
arXiv:2606. 15669v1 Announce Type: cross Abstract: Modern deep neural networks rely on Euclidean scalar activations (e.
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
arXiv:2607. 17822v1 Announce Type: cross Abstract: While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion.
Related stories
arXiv Machine Learning
Jul 24
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
Hugging Face Trending Papers
Jul 15
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.
arXiv AI
Jul 21
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
arXiv:2605. 29547v2 Announce Type: replace-cross Abstract: Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators.
arXiv Machine Learning
Aug 11
Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption
arXiv:2605. 29497v2 Announce Type: replace Abstract: We study the problem of robustly learning Gaussian Single Index Models (SIMs) in the presence of heavy-tailed noise and a constant fraction of adversarially corrupted covariates and responses.
arXiv Machine Learning
Jul 16
Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learning
arXiv:2605. 07914v2 Announce Type: replace Abstract: Sharpness-aware and gradient-alignment methods have been shown to improve generalization, however each family of methods targets a single geometric property of the loss landscape, while ignoring the other.
arXiv AI
Jul 16
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.
arXiv Machine Learning
Jun 15
Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
arXiv:2606. 14187v1 Announce Type: new Abstract: Large-scale neural network training increasingly relies on matrix-aware optimizers that exploit the structure of weight parameters beyond element-wise adaptation.
arXiv AI
Jun 16
ALCL: An Adaptive Log-Correntropy Loss for Robust Learning under Non-Gaussian Noise
arXiv:2606. 16050v1 Announce Type: cross Abstract: Robust deep learning under heavy-tailed and impulsive noise remains challenging because conventional losses such as mean squared error (MSE) exhibit unbounded sensitivity to outliers.
arXiv AI
5d ago
Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks
arXiv:2608. 12597v1 Announce Type: cross Abstract: Neural networks can often be trained or fine-tuned through random low-dimensional reparameterization, where a small latent vector is mapped into a full parameter update by a frozen random map.
arXiv AI
Jul 10
The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle
arXiv:2603. 17433v2 Announce Type: replace-cross Abstract: Transformer models have redefined sequence learning, yet dot-product self-attention introduces a quadratic token-mixing bottleneck for long-context time-series.
arXiv AI
Jul 14
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.