We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.
arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.
By Katie Everett
arXiv:2609.37680v1 Announce Type: cross
Abstract: One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can...
By Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba
arXiv:2604. 14037v2 Announce Type: replace Abstract: Parameter space is not function space for neural network architectures.
By Pranavkrishnan Ramakrishnan
arXiv:2608. 02816v1 Announce Type: new Abstract: We study the topology of learned representations in predictive coding networks (PCNs), a neuro-inspired bidirectional architecture, using a quantitative layer-wise persistent homology analysis.
By Adam Shaw, Jiayu Li, Michael Sperling, Michael Kim, Alvin Jin
arXiv:2609.39078v1 Announce Type: new
Abstract: Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and ar...
By Marvin Theiss, Lukas Braun, Andrew M. Saxe, Erin Grant
We study the topology of learned representations in predictive coding networks (PCNs), a neuro-inspired bidirectional architecture, using a quantitative layer-wise persistent homology analysis. We train well-performing PCNs on a synthetic classification dataset ($\geq 99.
The paper discusses tensorizing neural networks by reshaping dense weight matrices into higher-order tensors and approximating them with low-rank tensor network decompositions. This approach offers promising model compression and introduces bond indices that create new latent spaces, potentially enhancing interpretability. Despite encouraging empirical results, tensorized neural networks remain underused, and the authors call for more research to address practical scaling and adoption challenges.
By Safa Hamreras, Sukhbinder Singh, Rom\'an Or\'us
arXiv:2609.24379v1 Announce Type: cross
Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
By Gautam Ranka, Shubham Santosh Pandere, Aiden Dsouza
arXiv:2609.38081v1 Announce Type: new
Abstract: On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these...
By Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman, Leo Kozachkov, Kanaka Rajan
arXiv:2607. 05546v1 Announce Type: cross Abstract: We develop a unified function space theory of deep fully connected neural networks.
By Julia Nakhleh, Robert D. Nowak
The paper investigates how neural representations maintain the structure of input changes, linking representation analysis with internal interventions. It characterises when transformations can be applied through an encoder, providing linear settings where defects depend on discarded information and detailing failure modes for rectifiers and harmonic carriers. Using colour as a case study, the authors show that hue orbits in frozen visual features concentrate most energy in the first two harmonics, that this structure is inherited from input and architecture, and that a compact, fixed‑action interface can read hue zero‑shot with low error on unseen shapes.
By Yuan Sun