arXiv AI June 30, 2026 By Haoran Jin, Xiting Wang, Shijie Ren, Hong Xie, Defu Lian

C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

arXiv:2606. 30609v1 Announce Type: cross Abstract: Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 10

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.

By Weiduo Liao, Yunqiao Yang, Ying Wei

llms efficiency multimodal safety

arXiv Machine Learning

Jun 18

From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability

arXiv:2606. 18383v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable features from language models (LMs), yet a central question remains: when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM We study this through a post-hoc generalization framework that certifies the LM via a sparse proxy, obtained by replacing a native hidden activation with its pretrained SAE reconstruction.

By Dibyanayan Bandyopadhyay, Asif Ekbal

llms efficiency safety

arXiv Machine Learning

Jun 5

Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

arXiv:2606. 06333v1 Announce Type: new Abstract: Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional.

By Seyed Arshan Dalili, Mehrdad Mahdavi

llms efficiency safety

arXiv Machine Learning

Jun 15

Decompose Sparsely Where You Should, Absorb Densely Where You Should No

arXiv:2606. 14040v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are typically trained to reconstruct the \textbf{entire} residual stream through a sparse dictionary, implicitly assuming that all activation content is amenable to sparse, monosemantic decomposition.

By Ruixuan Deng, Zehao Jin, Zekun Wang, Zihan Dong

efficiency benchmarks safety

arXiv Machine Learning

Jun 17

Stable and Steerable Sparse Autoencoders with Weight Regularization

arXiv:2603. 04198v2 Announce Type: replace-cross Abstract: Sparse autoencoders (SAEs) are widely used to extract human-interpretable features from neural network activations, but their learned features can vary substantially across random seeds and training choices.

By Piotr Jedryszek, Oliver M. Crook

llms safety

arXiv Machine Learning

Jul 21

Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models

arXiv:2607. 17117v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence.

By Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik

llms

Related stories

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability

Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

Decompose Sparsely Where You Should, Absorb Densely Where You Should No

Stable and Steerable Sparse Autoencoders with Weight Regularization

Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models