arXiv AI
Jul 7
Rethinking Neural Nonlinearity as Gating
arXiv:2607. 03148v1 Announce Type: cross Abstract: Activation functions are considered an essential primitive for neural nonlinearity, i.
Rational Neural Networks have Expressivity Advantages
arXiv:2602. 12390v2 Announce Type: replace Abstract: We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax.
Related stories
arXiv Machine Learning
Jul 9
On Explicit Super-Expressive Approximation for Neural Networks
arXiv:2607. 06781v1 Announce Type: new Abstract: In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations.
Hugging Face Trending Papers
Jul 7
On Explicit Super-Expressive Approximation for Neural Networks
In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error.
arXiv AI
Jul 28
Exact and Asymptotically Complete Robust Verifications of Neural Networks via Ising Solvers
arXiv:2603. 00408v2 Announce Type: replace-cross Abstract: We present an Ising-compatible framework for formal neural-network robustness verification under bounded input perturbations.
arXiv AI
Jun 12
Structured vs. Unstructured Pruning: An Exponential Gap
arXiv:2603. 02234v3 Announce Type: replace-cross Abstract: The Strong Lottery Ticket Hypothesis (SLTH) states that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is sufficient.
arXiv AI
Jul 22
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
arXiv:2607. 18930v1 Announce Type: cross Abstract: The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error.
arXiv Machine Learning
Jun 24
Layer-wise Geometric Approximation Rates for Deep Networks
arXiv:2604. 20219v2 Announce Type: replace Abstract: Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear.
Hugging Face Trending Papers
Jul 21
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability.
arXiv Machine Learning
Aug 4
Optimized Piecewise Affine Abstractions of Neural Networks with Learnable Activation Functions
arXiv:2602. 06737v2 Announce Type: replace Abstract: We present a generalized framework for the range verification of neural networks featuring non-linear activation functions.
arXiv Machine Learning
Aug 4
Do Neural Networks Really Beat the Curse of Dimensionality? A Bit-Complexity View
arXiv:2608. 01357v1 Announce Type: new Abstract: Traditional approximation theory measures convergence rates in terms of the number of parameters or degrees of freedom.
arXiv Machine Learning
Jul 24
New Complexity-Theoretic Frontiers of Tractability for Neural Network Training
arXiv:2607. 20811v1 Announce Type: new Abstract: In spite of the fundamental role of neural networks in contemporary machine learning research, our understanding of the computational complexity of optimally training neural networks remains incomplete even when dealing with the simplest kinds of activation functions.
arXiv Machine Learning
Aug 11
Input convex neural networks as surrogates in mathematical optimisation
arXiv:2608. 09707v1 Announce Type: cross Abstract: Embedding trained neural networks as surrogates within optimisation problems is an established practice in operations research.