The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.
arXiv:2610.07389v1 Announce Type: cross Abstract: Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overl...
The paper introduces Consistent Sparse Dictionary Learning (ConSDL), a method that replaces the linear encoder in sparse autoencoders with per‑sample sparse coding to improve dictionary learning. Synthetic experiments show ConSDL achieves near‑perfect recovery of dictionaries and coefficients, and in real vision and language tasks it boosts cross‑seed consistency by 17.6% for dictionaries and 35.9% for coefficients compared to six SAE baselines.
arXiv:2606. 06333v1 Announce Type: new Abstract: Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional.
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv:2606. 14040v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are typically trained to reconstruct the \textbf{entire} residual stream through a sparse dictionary, implicitly assuming that all activation content is amenable to sparse, monosemantic decomposition.
arXiv:2606. 27802v1 Announce Type: new Abstract: Hierarchical predictive coding provides an interpretable framework for perception as error-driven inference in multi-layer generative models, while sparse coding imposes parsimonious latent representations through explicit sparsity constraints.
arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
The paper revisits parsimoniously activated dictionary learning (PADL), a method that imposes a global regularization on the number of activated dictionary atoms. It shows that PADL can be reformulated as a maximum a posteriori estimation under a structured generative model, enabling derivation of generalization guarantees and an analytical tradeoff between sparsity, storage cost, and reconstruction accuracy. Leveraging this insight, the authors propose an efficient PADL algorithm that removes manual hyperparameter tuning, improves reconstruction performance on visual benchmarks, and accelerates inference for vision‑language models.
The paper investigates when supervised learning can recover the true latent variables that generate data, focusing on two causal settings: generative (labels cause latents) and discriminative (latents cause labels). It shows that if the representation space has at least as many dimensions as the latent space, optimal encoders are linear; otherwise, identifiability fails and encoders become nonlinear. Additional constraints—such as non‑negative activations in the generative case or sparsity and spark conditions in the discriminative case—enable recovery of individual latent variables, linking nonlinear ICA, compressed sensing, and neural network interpretability.
arXiv:2607. 01799v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) decompose internal activations of neural networks into sparse linear combinations of learned features by fitting an overcomplete dictionary $\mathbf{W}\in\mathbb{R}^{m\times n}$ with $m<n$, and inferring a sparse code $\mathbf{x}\in\mathbb{R}^n$ from $\mathbf{h}\approx\mathbf{W}\mathbf{x}$.
arXiv:2606. 14990v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK.