arXiv Machine Learning
Jun 8
Sparsely gated tiny linear experts
arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.
Sparse Inter-Layer Dependencies of Transformer FFN Neurons
arXiv:2607. 11990v1 Announce Type: cross Abstract: Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream.
Related stories
arXiv Machine Learning
Sep 1
Tracing distinguishability through transformer processing with stochastic LayerNorm
arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...
arXiv AI
Jul 24
Scaling Interpretable Transformers with Parity Bottleneck Layers
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv AI
Jul 7
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
arXiv:2607. 02964v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does.
arXiv Machine Learning
Jul 31
Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
arXiv:2607. 27591v1 Announce Type: new Abstract: Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification.
arXiv AI
Jun 26
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
arXiv AI
Jun 16
DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
arXiv:2606. 15796v1 Announce Type: cross Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits.
arXiv AI
Jun 2
Rare Events, Real Signals: Functional Ensembles as Units of Computation in Deep Spiking Networks
arXiv:2606. 00073v1 Announce Type: cross Abstract: We investigate how internal representations emerge across hierarchical processing systems by introducing a neuroscience-inspired framework for analyzing deep spiking neural networks (SNN) through the lens of functional connectivity.
Hugging Face Trending Papers
Sep 21
Topographic Training Concentrates Causal Circuits Without Improving Neuron Monosemanticity
Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entangle many concepts in each neuron. Feature superposi...
arXiv Machine Learning
Sep 22
Topographic Training Concentrates Causal Circuits Without Improving Neuron Monosemanticity
arXiv:2609.24379v1 Announce Type: cross Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
arXiv AI
Jun 30
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.
arXiv Machine Learning
Aug 31
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
The paper introduces Mixture of Activations (MoA), a token‑adaptive feedforward network design that mixes multiple activation functions using lightweight gates while sharing linear projections. It also presents learnable activations (LA) as an input‑independent variant. The authors theoretically prove that MoA strictly surpasses both fixed‑activation FFNs and LA in expressive power, and empirically demonstrate that MoA achieves lower loss and better scaling on dense and MoE language models from 0.12 B to 2 B parameters with minimal overhead.