arXiv Machine Learning
Jun 30
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
arXiv:2606. 30444v1 Announce Type: cross Abstract: Neural networks are known to be susceptible to over-reliance on spurious correlations.
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations.
Related stories
arXiv AI
2d ago
Removing spurious minima for planar features by skip connections
arXiv:2610.01728v1 Announce Type: cross Abstract: Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple...
arXiv Machine Learning
Jun 5
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.
arXiv Machine Learning
Jun 30
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
arXiv AI
Jun 11
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
arXiv:2606. 12138v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs.
arXiv Machine Learning
Jun 18
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
arXiv:2603. 04895v2 Announce Type: replace-cross Abstract: Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima.
arXiv Machine Learning
Sep 23
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
arXiv Statistics ML
2d ago
Provable FDR Control for Deep Feature Selection: Deep MLPs and Beyond
arXiv:2512.04696v3 Announce Type: replace Abstract: We develop a flexible feature selection framework based on deep neural networks that approximately controls the false discovery rate (FDR), a measu...
arXiv Machine Learning
Jun 2
How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
arXiv Machine Learning
Jul 7
Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
arXiv:2607. 03613v1 Announce Type: new Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression.
arXiv Machine Learning
Jun 5
Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.