arXiv Machine Learning
Jun 8
Sparsely gated tiny linear experts
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
Language Model Circuits Are Sparse in the Neuron Basis
arXiv:2601. 22594v2 Announce Type: replace-cross Abstract: The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986).
Related stories
arXiv AI
Jun 16
Scalable Circuit Learning for Interpreting Large Language Models
arXiv:2606. 16939v1 Announce Type: cross Abstract: A prominent research direction in mechanistic interpretability is learning sparse circuits over LLM components to reveal how they jointly produce model behavior.
Hugging Face Trending Papers
6d ago
SAEVerbalizer: Generating Explanations for Sparse Autoencoder Features via Representation Verbalization
Sparse autoencoders (SAEs) are proposed to extract numerous features from large language model (LLM) representations, yet explaining these features still relies primarily on external observation. This reliance leads to superficial explanations inferred from observed model behavior and computational inefficiency from collecting such behavioral evidence at scale.
arXiv AI
Jun 16
DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
arXiv:2606. 15796v1 Announce Type: cross Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits.
arXiv AI
Jun 11
ICA Lens: Interpreting Language Models Without Training Another Dictionary
arXiv:2606. 11722v1 Announce Type: cross Abstract: Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior.
arXiv Machine Learning
Jul 2
Prototype Language Models
arXiv:2607. 00510v1 Announce Type: new Abstract: Knowing which training examples drive outputs is fundamental to auditing, correcting, and understanding language models, yet for modern LLMs this remains expensive, approximate, and largely post-hoc.
arXiv Machine Learning
Aug 7
CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
arXiv:2608. 05732v1 Announce Type: new Abstract: Controlling the behavior of large language models (LLMs) remains a critical challenge for AI alignment.
arXiv AI
Jun 26
Discovering Millions of Interpretable Features with Sparse Autoencoders
arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
arXiv Machine Learning
Jun 2
Step-Level Sparse Autoencoder for Reasoning Process Interpretation
arXiv:2603. 03031v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved strong complex reasoning capabilities through Chain-of-Thought (CoT) reasoning.
arXiv AI
Jul 24
Scaling Interpretable Transformers with Parity Bottleneck Layers
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv AI
Jul 7
InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
arXiv:2506. 07406v3 Announce Type: replace-cross Abstract: Understanding the internal representations of large language models (LLMs) is a central challenge in interpretability research.
arXiv AI
Jun 4
L$^3$: Large Lookup Layers
arXiv:2601. 21461v3 Announce Type: replace-cross Abstract: Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts.