The paper investigates how multilayer perceptrons (MLPs) learn features in regression tasks with clustered data. It finds that instead of forming a single global low‑dimensional representation, MLPs develop monosemantic specialized neurons—each neuron aligns strongly with a specific predictive feature relevant to a particular region of the input space. This specialization results in a collection of local low‑dimensional representations, giving MLPs a provable data‑efficiency advantage over methods that rely on a global representation.
By Amirhesam Abedsoltan, Enric Boix-Adsera, Fivos Kalogiannis, Mikhail Belkin
arXiv:2607. 23397v1 Announce Type: new Abstract: Hierarchical neural networks are widely used in artificial intelligence, yet their mathematical properties remain incompletely understood.
By Sumio Watanabe
arXiv:2606. 04583v1 Announce Type: new Abstract: Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.
By Ethem Alpaydin
arXiv:2610.23779v1 Announce Type: cross
Abstract: Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central p...
By Shu Zhou, Jean Barbier, Sebastian Goldt
The paper investigates how parameter symmetries in overparameterized nonlinear neural networks affect representational geometry. It shows that these symmetries can be described by three basic feature transformations—addition, duplication, and scaling—allowing a closed‑form decomposition of representations into essential and auxiliary parts. The study further demonstrates that implementation‑level selection rules can resolve the resulting degeneracy, producing identifiable geometries that reflect feature contributions to the network’s function.
By Marvin Theiss, Lukas Braun, Andrew M. Saxe, Erin Grant
arXiv:2606. 07007v1 Announce Type: cross Abstract: We propose a unified mathematical framework for a geometric understanding of concept learning and neuron interpretation in sparse autoencoders (SAEs).
By Chenhao Zhang, Chris Lin, Su-In Lee
arXiv:2109. 13445v3 Announce Type: replace-cross Abstract: The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the distribution of the training data is not well understood.
By Avi Cooper, Xavier Boix, Daniel Harari, Spandan Madan, Hanspeter Pfister, Tomotake Sasaki, Pawan Sinha
arXiv:2609.06862v1 Announce Type: new
Abstract: Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons...
By Dai Shi, Xiaoyu Li, Andi Han, Jos\'e Miguel Hern\'andez-Lobato
arXiv:2607. 11666v1 Announce Type: new Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization.
By Maksim A Kazanskii
arXiv:2608.20812v1 Announce Type: new
Abstract: We develop constructive approximation and learning guarantees for shallow neural models with infinite-dimensional inputs observed through finitely many...
By Pablo M. Bern\'a, Antonio Falc\'o, Diego Mond\'ejar
The paper introduces a pointwise generalization theory for fully connected deep neural networks, using a pointwise Riemannian Dimension derived from eigenvalues of learned feature representations across layers. This framework provides hypothesis-dependent, representation-aware generalization bounds that are significantly tighter than traditional size- or norm-based approaches, both theoretically and experimentally. The authors analytically identify structural properties that explain deep networks’ tractability and empirically show that the pointwise Riemannian Dimension captures feature compression, over‑parameterization effects, and optimizer bias.
By Shaojie Li, Yunbei Xu
arXiv:2607. 10430v1 Announce Type: cross Abstract: Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity.
By Hardik Rajpal, Dan Goodman