arXiv Statistics ML
High-probability guarantees for linear accessibility in feature superposition
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Jun 18
Effects of sparsity and superposition on loss in simple autoencoders
arXiv:2606. 18538v1 Announce Type: new Abstract: One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible for multiple different tasks, impeding a clean interpretation of their function.
arXiv Machine Learning
Sep 10
Feature Superposition in Neural Networks: From Theory to Practice
arXiv:2609.06862v1 Announce Type: new Abstract: Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons...
arXiv Machine Learning
Sep 25
Pointwise Generalization in Deep Neural Networks
The paper introduces a pointwise generalization theory for fully connected deep neural networks, using a pointwise Riemannian Dimension derived from eigenvalues of learned feature representations across layers. This framework provides hypothesis-dependent, representation-aware generalization bounds that are significantly tighter than traditional size- or norm-based approaches, both theoretically and experimentally. The authors analytically identify structural properties that explain deep networks’ tractability and empirically show that the pointwise Riemannian Dimension captures feature compression, over‑parameterization effects, and optimizer bias.
arXiv Machine Learning
Aug 31
Towards a mathematical theory of superposition
The paper develops a mathematical theory of superposition in neural networks using frame theory and compressed sensing. It shows that a sparse binary vector of active features can be encoded by an overcomplete dictionary and recovered via a ReLU operation with a suitable bias. The authors prove recovery theorems for both random-support and worst-case support settings, providing high-probability guarantees for low-coherence dictionaries and a sharp criterion for sparsity levels, with explicit results for Gaussian random matrices and equiangular tight frames.
arXiv Machine Learning
Jun 2
Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained models
arXiv:2503. 07325v2 Announce Type: replace Abstract: Understanding and certifying the behavior of modern deep neural networks remains a fundamental challenge in reliable machine learning.
arXiv Machine Learning
Oct 9
Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation
The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.