The paper develops a mathematical theory of superposition in neural networks using frame theory and compressed sensing. It shows that a sparse binary vector of active features can be encoded by an overcomplete dictionary and recovered via a ReLU operation with a suitable bias. The authors prove recovery theorems for both random-support and worst-case support settings, providing high-probability guarantees for low-coherence dictionaries and a sharp criterion for sparsity levels, with explicit results for Gaussian random matrices and equiangular tight frames.
arXiv:2509.01809v2 Announce Type: replace-cross Abstract: We consider the problem of support recovery for sparse binary signals from noisy linear measurements. For sparse Gaussian measurement matrice...
arXiv:2609.09556v1 Announce Type: new Abstract: Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear access...
arXiv:2602. 16568v2 Announce Type: replace-cross Abstract: Sparse recovery is among the most well-studied problems in learning theory and high-dimensional statistics.
arXiv:2608. 12982v1 Announce Type: new Abstract: In this research work, we are constructing the sensing matrix, which is essential for the success of the compressive sensing technique.
arXiv:2609. 02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and this dimension order is asymptotically optimal.
arXiv:2510. 24215v5 Announce Type: replace-cross Abstract: Recovery from linear measurements under sparse adversarial corruption is typically formulated as an exact-recovery problem: one seeks structural conditions on $\mathbf{A}$ (e.
arXiv:2606. 18538v1 Announce Type: new Abstract: One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible for multiple different tasks, impeding a clean interpretation of their function.
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
arXiv:2609.32176v2 Announce Type: replace Abstract: The support recovery problem in mixture of linear classifiers aims to identify the features relevant to the underlying decision rules when data is...
arXiv:2610.27010v1 Announce Type: cross Abstract: We study robust estimation of a Gaussian precision matrix that decomposes as a sparse matrix minus a symmetric, possibly indefinite, low-rank matrix,...
The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.
arXiv:2606. 14335v1 Announce Type: cross Abstract: Recovering structural information from noisy high-dimensional data is a fundamental task in statistical inference.