arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
By XinYang He, Wei Wang, Bing Zhao, Xuan Ren, WenBo Li, WeiXu Qiao, Hu Wei, Lin Qu
arXiv:2508. 17320v3 Announce Type: replace Abstract: Understanding the internal representations of large language models (LLMs) remains a central challenge for interpretability research.
By Yifei Yao, Hanrong Zhang, Mengnan Du
arXiv:2606. 14990v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK.
By Naiyu Yin, Yue Yu
Sparse autoencoders (SAEs) are commonly used to interpret large language models, but their reliability after pruning is unclear. This study shows that pruning’s effect on an SAE is governed by perturbation energy, a covariance-weighted norm, and that magnitude pruning distorts the representation space by ignoring activation geometry. Activation-aware pruning methods such as Wanda and SparseGPT better preserve SAE behavior, and the authors find that middle layers are especially vulnerable, leading them to propose a layer‑wise sparsity allocation that reduces perplexity for a given sparsity level.
By Suchit Gupte, Xueru Zhang, Mohammad Mahdi Khalili
arXiv:2606. 30609v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges.
By Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian
arXiv:2606. 03002v1 Announce Type: cross Abstract: Quantization is a standard path to deploying large language models, and a quantized model is typically judged acceptable when its perplexity or downstream accuracy stays close to the full-precision original.
By Evan Duan