The paper investigates the block-sparse featurizer (BSF), a model that uses small subspaces as atomic units instead of single directions, aiming to capture features on low-dimensional manifolds common in vision. It identifies that BSF still exhibits classic sparse autoencoder failure modes such as feature splitting and composition. The authors propose architectural improvements, notably a Tournament Top‑K selection rule, which markedly reduces feature splitting, and they extend the block concept to a crosscoder framework.
arXiv:2609.37857v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are increasingly scaled to wider dictionaries to recover fine-grained structure from large language model activations. Howev...
arXiv:2606. 27321v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features.
The paper introduces MASS, a hierarchical data selection method that first groups data using low‑dimensional manifold coordinates learned by a dense autoencoder, then applies a TopK sparse autoencoder for quality‑aware feature coverage within each group. This approach addresses the shortcomings of traditional diversity metrics that mix semantic, supervisory, and noise signals. Experiments on Vision Flan and LLaVA‑CoT demonstrate that MASS outperforms existing baselines across various budgets and can match or exceed full‑data training with only a small subset.
ShiftSplit-AD is a method that separates domain shift from defects in visual anomaly detection by decomposing the residual matrix of DINOv2 features into low‑rank and row‑sparse components. The sparse component is used for scoring anomalies, optionally fused with the low‑rank part. Experiments on AeBAD‑S show that sparse‑only scoring raises image AUROC from 0.6780 to 0.7294 and AUPRC from 0.8052 to 0.8465, but it also lowers clean AUROC on MVTec categories and hurts Bottle localization, highlighting a trade‑off between filtering shift and preserving defect information.
arXiv:2508. 16560v4 Announce Type: replace-cross Abstract: Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts.
arXiv:2608.28806v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) disentangle model activations into interpretable features and are widely used for steering large language models. Most exist...
arXiv:2505.22255v4 Announce Type: replace Abstract: Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent...
arXiv:2609.31620v1 Announce Type: new Abstract: Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual r...
arXiv:2012. 01982v3 Announce Type: replace Abstract: This paper proposes a standard way to represent sparse tensors.
Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning.
The paper introduces Dynamic DAE Guardrails (DSG), a method that uses Dynamic Sparse Autoencoders to perform precision unlearning in large language models. DSG leverages principled feature selection and a dynamic classifier to target activation-based unlearning, outperforming existing gradient‑based methods in terms of computational efficiency, stability, sequential unlearning, resistance to relearning attacks, data efficiency, and interpretability.
arXiv:2602. 14687v2 Announce Type: replace-cross Abstract: Improving Sparse Autoencoders (SAEs) requires benchmarks that can precisely validate architectural innovations.