arXiv:2606.22994v2 Announce Type: replace Abstract: Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces m...
arXiv:2609.36294v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) expose interpretable features in large language model activations, yet existing structured SAEs impose single-parent trees o...
arXiv:2606.16193v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representat...
arXiv:2606. 16193v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret.
Sparse autoencoders (SAEs) are usually trained on hundreds of millions to billions of activation tokens, but this study shows that the amount of data needed depends on the evaluation metric. Training four SAE architectures on three language models with token budgets from 5 M to 500 M revealed that reconstruction quality can reach near‑full‑data performance with far fewer tokens, while one‑to‑one feature reproducibility continues to improve as more data is used. The research also found that random sampling preserves activation distributions better for reconstruction, whereas diversity‑based sampling can enhance feature interpretability, and provided a theoretical framework distinguishing statistical consistency from the stricter conditions needed for exact feature recovery.
arXiv:2607. 17117v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence.
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
Persistent Sparse Autoencoders (Persistent SAEs) extend standard sparse autoencoders by learning a persistence coefficient for each feature, enabling the model to capture feature‑specific timescales from reconstruction alone. The study shows that these persistent features maintain competitive reconstruction quality while distinguishing between short‑timescale, locally interpretable features and long‑timescale, context‑accumulating features. In a prompt‑injection monitoring case study, slow features were found to preserve injection‑related signals and remain causally effective over long contexts.
SharedSAE demonstrates that a single sparse autoencoder can replace multiple model‑specific SAEs by using a shared dictionary with model‑specific encoder‑decoder pairs. It preserves activation magnitudes, normalizes only selection scores, and supports single‑model inference via model dropout. Trained on four 1B‑scale language models, SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, shows higher cross‑model latent correlations, and allows efficient adaptation of new models to the shared latent space.
arXiv:2606. 16939v1 Announce Type: cross Abstract: A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior.
arXiv:2508. 17320v3 Announce Type: replace Abstract: Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research.