The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.
By Vit\'oria Barin-Pacela, Shruti Joshi, Isabela Camacho, Simon Lacoste-Julien, David Klindt
arXiv:2610.07389v1 Announce Type: cross
Abstract: Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overl...
By Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, Miguel L\'azaro-Gredilla
The paper revisits parsimoniously activated dictionary learning (PADL), a method that imposes a global regularization on the number of activated dictionary atoms. It shows that PADL can be reformulated as a maximum a posteriori estimation under a structured generative model, enabling derivation of generalization guarantees and an analytical tradeoff between sparsity, storage cost, and reconstruction accuracy. Leveraging this insight, the authors propose an efficient PADL algorithm that removes manual hyperparameter tuning, improves reconstruction performance on visual benchmarks, and accelerates inference for vision‑language models.
By Zihui Zhao, Yuanbo Tang, Yang Li
arXiv:2606. 27941v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) provide useful decompositions of Transformer residual streams, but their learned features are usually named post hoc rather than directly connected to the Transformer's token vocabulary.
By Kairui Zhang, Ziwen Yu, Zahraa S. Abdallah, Martha Lewis
arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.
By Weiduo Liao, Yunqiao Yang, Ying Wei
The paper investigates Sparse Autoencoders (SAEs) by applying dictionary learning identifiability results to derive constraints on optimal dictionary learning features. It shows that optimal features cannot activate simultaneously, explaining SAE oddities such as hierarchical splitting, absorption, dense antipodal features, and infinite feature splitting as consequences of the dictionary learning objective. The derived constraints also serve as diagnostics, revealing that real SAEs perform well on training data but fail on increasingly distant test data.
By William Dorrell