arXiv Machine Learning
Jun 15
On the Geometry and Optimization of Polynomial Convolutional Networks
arXiv:2410. 00722v3 Announce Type: replace Abstract: We study convolutional neural networks with monomial activation functions.
Singular Learning and Occam's Razor in Deep Monomial Networks
arXiv:2606. 28464v1 Announce Type: new Abstract: In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture.
Related stories
arXiv Machine Learning
Aug 28
Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks
The paper proposes a conjecture that composing a fixed number of distinct nonconstant polynomials with a generic high‑degree polynomial produces linearly independent polynomials, extending Newman–Slater’s theorem. The authors prove the conjecture for two polynomials and for any number when the degrees are bounded, and they show how these results explain the parameter symmetries of deep fully connected neural networks with generic polynomial activations. In particular, for architectures with layer‑specific activations of increasing degree, the conjecture’s proven cases fully characterize the parameter sets that yield the same end‑to‑end network function, and it also resolves the identifiability of shallow polynomial networks.
arXiv Machine Learning
Jun 18
On the Stability of the Jacobian Matrix in Deep Neural Networks
arXiv:2506. 08764v3 Announce Type: replace Abstract: Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian.
arXiv Machine Learning
Sep 14
Benign Loss Landscapes Can Coexist with Worst-Case Hardness
The paper demonstrates that tree tensor networks (TTNs) can encode arbitrary read‑once Boolean formulas, yielding polynomial‑size targets that are hard for gradient descent to learn in polynomial time, yet their loss landscapes are conditionally benign: every minimum‑norm local minimum is global. This shows that bad local minima are not the source of learning difficulty in TTNs; instead, high‑order degenerate saddle points caused by rank‑deficiency can impede learning. A case study on the parity function illustrates how TTNs can link landscape geometry to computational hardness.
arXiv Machine Learning
Jul 28
A Statistical Difference between Single-Layer Learning and Hierarchical Learning in Wide Neural Networks
arXiv:2607. 23397v1 Announce Type: new Abstract: Hierarchical neural networks are widely used in artificial intelligence, yet their mathematical properties remain incompletely understood.
arXiv Machine Learning
Jun 19
Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defects
arXiv:2605. 09609v2 Announce Type: replace Abstract: We provide counterexamples to the unimodal minimal filling architecture conjecture for polynomial neural networks (PNNs) with power activation functions.
arXiv Machine Learning
Jun 10
Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks
arXiv:2603. 12785v2 Announce Type: replace Abstract: Three-layer neural networks are known to form singular learning models, and their Bayesian asymptotic behavior is governed by the learning coefficient, or real log canonical threshold.
arXiv Machine Learning
Jul 10
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
arXiv:2607. 07884v1 Announce Type: new Abstract: In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth.
arXiv Machine Learning
Aug 11
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
arXiv:2502. 11152v4 Announce Type: replace-cross Abstract: The optimization foundations of deep linear networks have recently received significant attention.
arXiv Machine Learning
Jul 14
Singular perturbations and hierarchical learning in two-layer neural networks
arXiv:2607. 10869v1 Announce Type: new Abstract: We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension.
arXiv AI
Jun 4
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
arXiv:2606. 04327v1 Announce Type: cross Abstract: We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions.
arXiv Machine Learning
Aug 11
Correlation flow governs learning at criticality
arXiv:2608. 08350v1 Announce Type: new Abstract: The initialisation of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive.