arXiv Machine Learning
Shaped SAEs: Expanding the Toolkit for Mechanistic Interpretability
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 2
How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations
arXiv:2606. 02385v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control.
arXiv AI
Jun 26
Discovering Millions of Interpretable Features with Sparse Autoencoders
arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
arXiv AI
Jun 11
ICA Lens: Interpreting Language Models Without Training Another Dictionary
arXiv:2606. 11722v1 Announce Type: cross Abstract: Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior.
arXiv Machine Learning
Jun 5
Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
arXiv:2606. 06333v1 Announce Type: new Abstract: Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional.
arXiv AI
Oct 15
On the Sparsity-Storage-Accuracy Tradeoff in Parsimoniously Activated Dictionary Learning
The paper revisits parsimoniously activated dictionary learning (PADL), a method that imposes a global regularization on the number of activated dictionary atoms. It shows that PADL can be reformulated as a maximum a posteriori estimation under a structured generative model, enabling derivation of generalization guarantees and an analytical tradeoff between sparsity, storage cost, and reconstruction accuracy. Leveraging this insight, the authors propose an efficient PADL algorithm that removes manual hyperparameter tuning, improves reconstruction performance on visual benchmarks, and accelerates inference for vision‑language models.
arXiv AI
Oct 19
Beyond SAEs: Consistent Concept Discovery with Sparse Coding
The paper introduces Consistent Sparse Dictionary Learning (ConSDL), a method that replaces the linear encoder in sparse autoencoders with per‑sample sparse coding to improve dictionary learning. Synthetic experiments show ConSDL achieves near‑perfect recovery of dictionaries and coefficients, and in real vision and language tasks it boosts cross‑seed consistency by 17.6% for dictionaries and 35.9% for coefficients compared to six SAE baselines.