arXiv:2607. 17117v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence.
By Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik
SharedSAE demonstrates that a single sparse autoencoder can replace multiple model‑specific SAEs by using a shared dictionary with model‑specific encoder‑decoder pairs. It preserves activation magnitudes, normalizes only selection scores, and supports single‑model inference via model dropout. Trained on four 1B‑scale language models, SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, shows higher cross‑model latent correlations, and allows efficient adaptation of new models to the shared latent space.
By Daniil Ognev, C\'elian Vasson, Lijie Hu, Kentaro Inui, Benjamin Heinzerling
Sparse autoencoders (SAEs) are usually trained on hundreds of millions to billions of activation tokens, but this study shows that the amount of data needed depends on the evaluation metric. Training four SAE architectures on three language models with token budgets from 5 M to 500 M revealed that reconstruction quality can reach near‑full‑data performance with far fewer tokens, while one‑to‑one feature reproducibility continues to improve as more data is used. The research also found that random sampling preserves activation distributions better for reconstruction, whereas diversity‑based sampling can enhance feature interpretability, and provided a theoretical framework distinguishing statistical consistency from the stricter conditions needed for exact feature recovery.
By Sambit Ghosh, Madelyn Mathai, R. Venkatesh Babu, Chirag Agarwal
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv:2606. 30609v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges.
By Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian
arXiv:2606. 18383v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable features from language models (LMs), yet a central question remains: when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM We study this through a post-hoc generalization framework that certifies the LM via a sparse proxy, obtained by replacing a native hidden activation with its pretrained SAE reconstruction.
By Dibyanayan Bandyopadhyay, Asif Ekbal