arXiv Machine Learning June 18, 2026 By Mriganka Basu Roy Chowdhury, Eric McLaughlin Weiner

Effects of sparsity and superposition on loss in simple autoencoders

Read the original on arXiv Machine Learning →

arXiv:2606. 18538v1 Announce Type: new Abstract: One of the major difficulties in the mechanistic interpretability of neural networks is the occurrence of polysemanticity, which suggests that each neuron is typically responsible for multiple different tasks, impeding a clean interpretation of their function.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 7

Compressed Computation under $L^4$ Loss is likely Computation in Superposition

arXiv:2607. 04800v1 Announce Type: new Abstract: Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions.

By Francisco Ferreira da Silva, Stefan Heimersheim

arXiv Machine Learning

Jun 5

Expand Neurons, Not Parameters

arXiv:2510. 04500v3 Announce Type: replace Abstract: This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance.

By Linghao Kong, Inimai Subramanian, Yonadav Shavit, Micah Adler, Dan Alistarh, Nir Shavit

rag safety

arXiv Machine Learning

Aug 10

Equivariant Sparse Autoencoders: Mechanistic Interpretability of Neural Networks on Symmetric Data

arXiv:2511. 09432v2 Announce Type: replace Abstract: Machine learning (ML) models achieve remarkable performance but remain hard to interpret due to their scale and complexity.

By Ege Erdogan, Ana Lucic

safety

arXiv Machine Learning

Jun 24

Similarity of Neural Network Representations in Superposition

arXiv:2604. 00208v2 Announce Type: replace Abstract: Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features.

By Sunny Liu, Habon Issa, Andr\'e Longon, Liv Gorton, Meenakshi Khosla, Alex Williams, David Klindt

llms safety

arXiv AI

Jul 10

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

arXiv:2607. 08605v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept.

By Weiduo Liao, Yunqiao Yang, Ying Wei

llms efficiency multimodal safety

arXiv AI

Jul 8

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

arXiv:2508. 16560v4 Announce Type: replace-cross Abstract: Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts.

By David Chanin, Adri\`a Garriga-Alonso

llms efficiency

Related stories

Compressed Computation under $L^4$ Loss is likely Computation in Superposition

Expand Neurons, Not Parameters

Equivariant Sparse Autoencoders: Mechanistic Interpretability of Neural Networks on Symmetric Data

Similarity of Neural Network Representations in Superposition

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders