arXiv:2602. 19799v2 Announce Type: replace-cross Abstract: Despite recent algorithmic advances, we still lack principled ways to leverage the well-documented rescaling symmetries in ReLU neural network parameters.
arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.
The paper studies how grouping Layer Normalization (LN) units influences the approximation capacity of neural networks as their width increases. It shows that a shallow LN-Net with a single LN unit cannot approximate arbitrary functions regardless of width, but Parallel Layer Normalization networks (PLN-Nets) become universal as the number of parallel groups increases. The authors extend this result to deep networks, RMSNorm variants, feed‑forward and convolutional architectures, and validate the practical benefits of PLN through controlled MLP and physics‑informed neural network experiments, while also demonstrating that group count, grouped axes, replacement site, and label structure jointly affect fitting and generalization in CNN and ViT settings.
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
arXiv:2509.00362v2 Announce Type: replace Abstract: Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause per...
arXiv:2607. 08406v1 Announce Type: new Abstract: Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients.
arXiv:2510. 01175v2 Announce Type: replace Abstract: While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited.
The paper introduces a new weight update method for morphological neural networks based on the Multiplicative Weights Update scheme. It treats each morphological perceptron as a learning-from-experts problem in logarithmic space, using a correlation-based reward that favors inputs aligned with the desired output change even when gradient signals are weak. Experiments on fully connected layers and transformer components across nine benchmarks show that correlational training improves performance on eight tasks, up to 32.84 percentage points, and reduces run‑to‑run variability.
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence.
arXiv:2606. 02526v1 Announce Type: cross Abstract: Long-tailed recognition poses a significant challenge for deep learning.
The paper introduces a new weight update method for morphological neural networks based on the Multiplicative Weights Update scheme. It treats each morphological perceptron as a learning-from-experts problem in logarithmic space and uses a correlation-based reward that favors inputs aligned with the desired output change, even when gradient signals are weak. Experiments on fully connected layers and transformer components across nine benchmarks show that correlational training improves performance on eight tasks, up to 32.84 percentage points, and reduces run‑to‑run variability.
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.