The paper presents a near-complete, nonasymptotic generalization theory for multilayer neural networks using path regularization, applicable to broad Lipschitz loss functions without requiring bounded loss or extreme network hyperparameters. It provides an explicit upper bound that addresses approximation rates in generalized Barron spaces and demonstrates the double descent phenomenon for ReLU networks. The authors claim near-minimax optimality for regression problems and plan to establish matching lower bounds in future work.
By Hao Yu
arXiv:2610. 25116v2 Announce Type: replace Abstract: We study positive, bias-free, one-hidden-layer ReLU networks under standard Gaussian inputs.
By Zhijie Zheng
The paper presents a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, covering both underparameterized and overparameterized regimes without assuming bounded loss, sparsity, or width/depth limits. It explicitly models approximation error, resolves an open problem on approximation rates in generalized Barron spaces, and establishes matching upper and lower bounds that show the error curve is non‑monotone—descending, rising, then descending again—as width increases. The analysis reveals that the rise is due to shrinkage imposed by regularization rather than overfitting, and this effect cannot be eliminated by width‑independent regularization.
By Hao Yu
arXiv:2505. 18113v2 Announce Type: replace Abstract: Training quantized neural networks requires addressing the non-differentiable and discrete nature of the underlying optimization problem.
By Halyun Jeong, Jack Xin, Penghang Yin
arXiv:2608. 11479v1 Announce Type: new Abstract: We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset.
By Siqiao Mu, Diego Klabjan
arXiv:2609. 07997v1 Announce Type: new Abstract: We characterize the sharp structure-agnostic minimax risk for coefficient estimation in the partial linear model when the outcome and treatment nuisances are learned by two distinct black-box learners, which resolves the open problem in double machine learning posed by Gu (2025).
By Haichen Hu, David Simchi-Levi
arXiv:2610. 23007v1 Announce Type: new Abstract: Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated.
By Quan Nguyen, Robin Yadav, Param Mody, Mark Schmidt
The paper introduces a method for adversarial training that avoids computing input gradients by using a low‑rank Householder expansion (LRHE) to directly generate small‑norm adversarial examples from a network’s parameters. This approach requires only forward passes and standard back‑propagation, eliminating the inner maximization loop and reducing computational cost to roughly 2.8 PGD steps per epoch. The resulting models achieve comparable robustness to multi‑step PGD training for small relative ε budgets, demonstrating the feasibility of gradient‑free adversarial training.
By Tiana C. Johnson, Donsub Rim
arXiv:2606. 06934v1 Announce Type: new Abstract: We analyze generalization error, uniform stability, and uniform argument stability of gradient descent (GD) and stochastic gradient descent (SGD) over discrete parameter spaces, where each update involves deterministic or stochastic rounding.
By Jonghyun Shin, Sejun Park
The paper investigates how learning algorithms behave when gradient signals are noisy, sparse, and intermittently available, as often occurs in biological neural networks. It demonstrates that even if the average learning dynamics converge to the correct optimum, the variance of learning can become asymptotically infinite, leading to instability. The authors analyze several analytically tractable settings—including single-layer and multi-layer linear networks—to derive explicit conditions under which learning remains stable, highlighting that unbiased credit assignment alone is insufficient without sufficiently frequent and informative feedback.
By John J. Vastola
arXiv:2609.07755v1 Announce Type: new
Abstract: Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training d...
By Yuqing Wang, Ioannis G. Kevrekidis, Mikhail Belkin
The paper introduces Curvature-Scaled Sharpness-Aware Minimization (CS‑SAM), a variant of SAM that scales both the perturbation and descent step by curvature bounds to uniformly control gradient distortion under generalized smoothness conditions. It proves deterministic ≥1/T convergence rates for nonconvex objectives, last‑iterate ≥1/T rates for convex objectives, and linear convergence under the Polyak‑Łojasiewicz condition, while recovering standard gradient descent rates when the perturbation is zero. Experiments on computer vision tasks show that CS‑SAM, particularly the (H0,H1) variant, achieves the highest mean best‑test accuracy across four settings compared to tuned SAM and USAM.
By Dimitris Oikonomou, Nicolas Loizou