arXiv Machine Learning
Jun 30
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
arXiv:2606. 30444v1 Announce Type: cross Abstract: Neural networks are known to be susceptible to over-reliance on spurious correlations.
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations.
Related stories
arXiv Machine Learning
Jun 5
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.
arXiv Machine Learning
Jun 30
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
arXiv AI
Jun 11
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
arXiv:2606. 12138v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs.
arXiv Machine Learning
Jun 18
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
arXiv:2603. 04895v2 Announce Type: replace-cross Abstract: Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima.
arXiv Machine Learning
Jun 2
How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
arXiv Machine Learning
Jul 7
Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
arXiv:2607. 03613v1 Announce Type: new Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression.
arXiv Machine Learning
Jun 5
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Jun 18
Effects of sparsity and superposition on loss in simple autoencoders
arXiv:2606. 18538v1 Announce Type: new Abstract: One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible for multiple different tasks, impeding a clean interpretation of their function.
arXiv Machine Learning
Jun 5
Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv Machine Learning
Jun 17
Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks
arXiv:2606. 17120v1 Announce Type: new Abstract: Deep neural networks (DNNs) exhibit first order phase transitions under variations of the L2 regularization strength, with each transition marking the onset of a new learnable feature.