arXiv AI
Inference and learning in sparse autoencoders as natural gradient flow
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Oct 9
Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation
The paper argues that the linear representation hypothesis—where neural activations encode concepts as linear mixtures—breaks down under superposition, causing linear decision boundaries to lose linearity after projection. Classical sparse coding with per‑sample inference can recover latent factors, but sparse autoencoders (SAEs) amortise this inference into a fixed encoder, creating a systematic gap that persists across training sizes, latent dimensions, and sparsity levels. Experiments show that the failure of SAEs under out‑of‑distribution compositional shifts stems from poor dictionary learning rather than inference, and that a good dictionary can solve the problem at all tested scales, highlighting scalable dictionary learning as the key open challenge.
arXiv AI
1d ago
Beyond SAEs: Consistent Concept Discovery with Sparse Coding
The paper introduces Consistent Sparse Dictionary Learning (ConSDL), a method that replaces the linear encoder in sparse autoencoders with per‑sample sparse coding to improve dictionary learning. Synthetic experiments show ConSDL achieves near‑perfect recovery of dictionaries and coefficients, and in real vision and language tasks it boosts cross‑seed consistency by 17.6% for dictionaries and 35.9% for coefficients compared to six SAE baselines.
Hugging Face Trending Papers
Aug 13
SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv Machine Learning
Sep 7
SharedSAE: One Feature Dictionary Across Language Models
SharedSAE demonstrates that a single sparse autoencoder can replace multiple model‑specific SAEs by using a shared dictionary with model‑specific encoder‑decoder pairs. It preserves activation magnitudes, normalizes only selection scores, and supports single‑model inference via model dropout. Trained on four 1B‑scale language models, SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, shows higher cross‑model latent correlations, and allows efficient adaptation of new models to the shared latent space.
arXiv AI
Jul 24
Scaling Interpretable Transformers with Parity Bottleneck Layers
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv AI
Jun 11
Cross-Layer Discrete Concept Discovery for Interpreting Language Models
arXiv:2506. 20040v3 Announce Type: replace-cross Abstract: Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure.