The paper investigates the relationship between representation and function in neural networks, using analytical two‑layer linear models and simulations of nonlinear networks. It demonstrates that functional similarity and representational similarity can be dissociated: networks may share representations without sharing functions, and vice versa. The study also finds that robustness to input noise or generalization error does not constrain representations, whereas robustness to parameter noise forces networks to adopt task‑specific representations, indicating that representational alignment reflects computational advantages beyond mere functional alignment.
By Lukas Braun, Erin Grant, Andrew M. Saxe
arXiv:2511. 09432v2 Announce Type: replace Abstract: Machine learning (ML) models achieve remarkable performance but remain hard to interpret due to their scale and complexity.
By Ege Erdogan, Ana Lucic
arXiv:2606. 10913v1 Announce Type: new Abstract: We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks.
By Jakob Galley, Vahid Shahverdi, Axel Flinth
The paper investigates how parameter symmetries in overparameterized nonlinear neural networks affect representational geometry. It shows that these symmetries can be described by three basic feature transformations—addition, duplication, and scaling—allowing a closed‑form decomposition of representations into essential and auxiliary parts. The study further demonstrates that implementation‑level selection rules can resolve the resulting degeneracy, producing identifiable geometries that reflect feature contributions to the network’s function.
By Marvin Theiss, Lukas Braun, Andrew M. Saxe, Erin Grant
arXiv:2608. 08309v1 Announce Type: cross Abstract: We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy.
By Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki
arXiv:2604. 00208v2 Announce Type: replace Abstract: Comparing internal representations is a central goal in neuroscience and machine learning, but standard linear alignment metrics (Representational Similarity Analysis, Centered Kernel Alignment, and linear regression) are frequently applied to neural activity coordinates rather than on the underlying features.
By Sunny Liu, Habon Issa, Andr\'e Longon, Liv Gorton, Meenakshi Khosla, Alex Williams, David Klindt
The paper investigates how finite‑stepsize stochastic gradient descent (SGD) breaks scale symmetry in undercomplete linear autoencoders. It shows that SGD induces a directed scale drift that favors large decoder weights on the PCA solution manifold, a process that is slow and analytically tractable but eventually halted by a stability boundary. The resulting solutions exhibit sharper loss Hessians compared to a balanced baseline, illustrating how loss geometry can transform residual gradient noise into directed motion along equivalent solution manifolds.
By Farhad Pashakhanloo, Jacob A. Zavatone-Veth
arXiv:2606. 04754v1 Announce Type: new Abstract: Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged.
By Vincent B\"urgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka
arXiv:2610.17424v1 Announce Type: cross
Abstract: Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional neural networks...
By Tomas Karella, Emily Shinkle, Alice Elisabeth Anastasia Allen, Pieter J. Swart, Nicholas Lubbers, Roxana Bujack
The paper introduces graph metanetworks for transformer-based language models, addressing the challenge that different weight configurations can implement the same function due to a rich symmetry group. It formalizes a weight‑space symmetry group for complete decoder‑only language models and presents a graph construction invariant to these symmetries. The authors also create two new transformer‑population datasets with tasks like training‑corpus prediction and fine‑tuning detection, where their method outperforms existing baselines.
By Yonathan Wolanowsky, Guy Bar-Shalom, Fabrizio Frasca, Haggai Maron
Artificial neural networks generate local symmetries called fibrations and coverings during learning, and these covering symmetries are stable attractors of stochastic gradient descent. The study shows that such symmetries appear across diverse architectures—multilayer, convolutional, recurrent, and transformer networks—and can be exploited for drastic model compression, reducing networks to 17% of their original size without performance loss. Controlled breaking of covering symmetry further improves continual learning, achieving state‑of‑the‑art results.
By Osvaldo M Velarde, Lucas C Parra, Alireza Hashemi, Hernan A Makse
arXiv:2512. 20963v3 Announce Type: replace Abstract: Diffusion models excel at generating high-quality, diverse samples, yet they risk memorizing training data when overfit to the training objective.
By Zekai Zhang, Xiao Li, Xiang Li, Lianghe Shi, Meng Wu, Molei Tao, Qing Qu