Persistent Sparse Autoencoders (Persistent SAEs) extend standard sparse autoencoders by learning a persistence coefficient for each feature, enabling the model to capture feature‑specific timescales from reconstruction alone. The study shows that these persistent features maintain competitive reconstruction quality while distinguishing between short‑timescale, locally interpretable features and long‑timescale, context‑accumulating features. In a prompt‑injection monitoring case study, slow features were found to preserve injection‑related signals and remain causally effective over long contexts.
By Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik
SharedSAE demonstrates that a single sparse autoencoder can replace multiple model‑specific SAEs by using a shared dictionary with model‑specific encoder‑decoder pairs. It preserves activation magnitudes, normalizes only selection scores, and supports single‑model inference via model dropout. Trained on four 1B‑scale language models, SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, shows higher cross‑model latent correlations, and allows efficient adaptation of new models to the shared latent space.
By Daniil Ognev, C\'elian Vasson, Lijie Hu, Kentaro Inui, Benjamin Heinzerling
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv:2606. 30609v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges.
By Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian
arXiv:2508. 17320v3 Announce Type: replace Abstract: Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research.
By Yifei Yao, Hanrong Zhang, Mengnan Du
arXiv:2609.07876v1 Announce Type: cross
Abstract: Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linea...
By Arjun Patrawala, Jiahai Feng, Erik Jones, Jacob Steinhardt