The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.
By Vit\'oria Barin-Pacela, Shruti Joshi, Isabela Camacho, Simon Lacoste-Julien, David Klindt
arXiv:2610.07389v1 Announce Type: cross
Abstract: Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overl...
By Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, Miguel L\'azaro-Gredilla
The paper revisits parsimoniously activated dictionary learning (PADL), a method that imposes a global regularization on the number of activated dictionary atoms. It shows that PADL can be reformulated as a maximum a posteriori estimation under a structured generative model, enabling derivation of generalization guarantees and an analytical tradeoff between sparsity, storage cost, and reconstruction accuracy. Leveraging this insight, the authors propose an efficient PADL algorithm that removes manual hyperparameter tuning, improves reconstruction performance on visual benchmarks, and accelerates inference for vision‑language models.
By Zihui Zhao, Yuanbo Tang, Yang Li
arXiv:2606. 27941v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) provide useful decompositions of Transformer residual streams, but their learned features are usually named post hoc rather than directly connected to the Transformer's token vocabulary.
By Kairui Zhang, Ziwen Yu, Zahraa S. Abdallah, Martha Lewis
arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.
By Weiduo Liao, Yunqiao Yang, Ying Wei
The paper investigates Sparse Autoencoders (SAEs) by applying dictionary learning identifiability results to derive constraints on optimal dictionary learning features. It shows that optimal features cannot activate simultaneously, explaining SAE oddities such as hierarchical splitting, absorption, dense antipodal features, and infinite feature splitting as consequences of the dictionary learning objective. The derived constraints also serve as diagnostics, revealing that real SAEs perform well on training data but fail on increasingly distant test data.
By William Dorrell
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
By William Dorrell
SharedSAE demonstrates that a single sparse autoencoder can replace multiple model‑specific SAEs by using a shared dictionary with model‑specific encoder‑decoder pairs. It preserves activation magnitudes, normalizes only selection scores, and supports single‑model inference via model dropout. Trained on four 1B‑scale language models, SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, shows higher cross‑model latent correlations, and allows efficient adaptation of new models to the shared latent space.
By Daniil Ognev, C\'elian Vasson, Lijie Hu, Kentaro Inui, Benjamin Heinzerling
arXiv:2606. 30609v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges.
By Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian
arXiv:2607. 17117v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence.
By Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik
arXiv:2508. 17320v3 Announce Type: replace Abstract: Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research.
By Yifei Yao, Hanrong Zhang, Mengnan Du
arXiv:2509. 10033v2 Announce Type: replace Abstract: Sparse dictionary coding represents signals as linear combinations of a few dictionary atoms.
By Boya Ma, Abram Magner, Maxwell McNeil, Petko Bogdanov