arXiv Machine Learning
Jun 9
Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics
arXiv:2606. 09744v1 Announce Type: new Abstract: We study feed-forward ReLU networks with fixed readout and quadratic loss.
Network Dynamics-Based Framework for Understanding Deep Neural Networks
arXiv:2501. 02436v5 Announce Type: replace Abstract: Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning.
Related stories
arXiv Machine Learning
Jul 3
From Approximation to Emergence: A Theory of Deep Learning
arXiv:2607. 01311v1 Announce Type: new Abstract: Deep learning has outgrown any single mathematical explanation.
arXiv Machine Learning
Jul 24
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
arXiv:2607. 21005v1 Announce Type: new Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable.
arXiv AI
Jun 29
Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning
arXiv:2501. 07400v2 Announce Type: replace-cross Abstract: We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations.
arXiv Machine Learning
Jul 7
Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?
arXiv:2511. 07308v3 Announce Type: replace Abstract: Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights.
Hugging Face Trending Papers
Sep 8
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
The paper investigates how normalization makes neural networks scale‑invariant, creating a feedback loop between learning‑rate schedules and weight decay that controls the effective step size of the optimizer. It derives an exact discrete‑time law showing that a single scalar quantity captures all schedule and decay effects, with norm growth providing a self‑quenching counter‑force that defines a sharp boundary between contraction‑ and expansion‑dominated regimes. Through exact analysis of a normalized regression model and experiments on MLPs, CNNs, GPT‑2, and various datasets, the authors demonstrate that constant learning rates with weight decay are intrinsically unstable, leading to recurrent dynamics, and that adaptive optimizers exhibit weaker stabilization under normalization. "whyItMatters":"The study provides a precise, actionable rule for controlling training dynamics and schedule design in modern deep learning by isolating a single governing quantity for scale‑invariant optimization."
arXiv Machine Learning
Jun 9
Bulk-boundary decomposition of neural networks
arXiv:2511. 02003v2 Announce Type: replace Abstract: We present the bulk--boundary decomposition as a new framework for understanding the training dynamics of deep neural networks.
arXiv Machine Learning
Jun 17
A Dynamical Systems Perspective on the Analysis of Neural Networks
arXiv:2507. 05164v2 Announce Type: replace-cross Abstract: In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms.
arXiv Machine Learning
Jun 30
Scalar Representations of Neural Network Training Dynamics
arXiv:2606. 30384v1 Announce Type: new Abstract: Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape.
Hugging Face Trending Papers
Jun 29
Scalar Representations of Neural Network Training Dynamics
Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging.
arXiv AI
Jul 7
The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System
arXiv:2607. 04993v1 Announce Type: cross Abstract: Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them.
arXiv Machine Learning
Jun 24
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
arXiv:2311. 02960v5 Announce Type: replace Abstract: Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data.