How Optimality Structures Sparse Dictionaries: Theory for Interpreting SAE Representations

The paper investigates Sparse Autoencoders (SAEs) by applying dictionary learning identifiability results to derive constraints on optimal dictionary learning features. It shows that optimal features cannot activate simultaneously, explaining SAE oddities such as hierarchical splitting, absorption, dense antipodal features, and infinite feature splitting as consequences of the dictionary learning objective. The derived constraints also serve as diagnostics, revealing that real SAEs perform well on training data but fail on increasingly distant test data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.