arXiv:2607. 18930v1 Announce Type: cross Abstract: The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error.
The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability.
arXiv:2606. 04327v1 Announce Type: cross Abstract: We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions.
arXiv:2610. 31264v1 Announce Type: new Abstract: We construct a continuous function on $[-1,1]^2$ for which the infimum of the $L^2$-approximation error over the class of ReLU networks with two hidden layers and architecture $(2,2,1,1)$ is not attained.
The paper investigates how multilayer perceptrons (MLPs) learn features in regression tasks with clustered data. It finds that instead of forming a single global low‑dimensional representation, MLPs develop monosemantic specialized neurons—each neuron aligns strongly with a specific predictive feature relevant to a particular region of the input space. This specialization results in a collection of local low‑dimensional representations, giving MLPs a provable data‑efficiency advantage over methods that rely on a global representation.
The paper introduces a new family of concentration metrics, called α-RC, based on Rényi divergence to assess how the magnitudes of Jacobian minors in multilayer perceptrons are distributed. Unlike the existing Participation Ratio, these metrics are bounded between 0 and 1 regardless of network size, with 0 meaning uniform spread and 1 indicating dominance by a single minor. For α=2, the α-RC metric reduces to a transformed version of the Participation Ratio, and experiments on single-hidden-layer MLPs demonstrate its practical advantages.
arXiv:2608.24007v1 Announce Type: new Abstract: Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has...
arXiv:2610.31783v1 Announce Type: new Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
arXiv:2609.24379v1 Announce Type: cross Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
The paper investigates how the geometry of teacher neural networks affects the learnability of student networks in teacher‑student setups. By formalizing learnability as the success rate of reaching the global minimum, the authors identify two teacher distributions—one maximizing node dissimilarity (easy) and one minimizing it (hard)—that lead to markedly different success rates across various settings and activation functions. They analyze the loss landscape of small networks, revealing two types of suboptimal local minima (out‑of‑bounds and interior) whose attraction regions depend on teacher structure, and demonstrate that adjusting learning rates for the readout layer and inner biases can improve success rates. whyItMatters:"The study highlights that teacher geometry, often overlooked, plays a crucial role in determining how effectively a student network can learn, offering guidance for designing more realistic teacher‑student experiments."
The paper introduces a new neural network architecture that represents mixed‑monotone functions as the difference of two monotone subnetworks, enabling tight output enclosures via only two forward passes of opposite corners of an input interval. The enclosure size is a closed‑form function of the network weights, allowing direct minimization during training. Experiments on regression and dynamical‑system benchmarks show that the trained networks achieve enclosure sizes within 0.5% of a proven lower bound while maintaining prediction accuracy, and the method is significantly faster and often tighter than existing techniques such as CROWN, α‑CROWN, and zonotope propagation.