The paper analyzes Gradient Descent (GD) on a minimal binary classification problem with a two‑neuron ReLU network trained on two instances. It proves that GD converges to an optimal robustness margin—maximizing the distance between the decision boundary and training points—but does so at a very slow rate, specifically θ(1/ln(t)). Empirical simulations confirm this slow convergence across various initializations, and the authors provide a rigorous analysis of GD trajectories across different activation patterns to establish the first explicit lower bound on robustness convergence in a non‑linear model.
By Guy Smorodinsky, Sveta Gimpleson, Itay Safran
arXiv:2410. 07719v4 Announce Type: replace Abstract: Despite being widely adopted as a canonical framework for learning robust models, adversarial training suffers from robust overfitting.
By Yuelin Xu, Xiao Zhang
arXiv:2607. 00207v1 Announce Type: cross Abstract: We develop a framework for analyzing the learning dynamics of $\ell_2$-adversarial training of single-index models on Gaussian mixtures in the high-dimensional limit under streaming stochastic gradient descent (SGD).
By Fabrizzio Sabelli
arXiv:2608. 14594v1 Announce Type: new Abstract: Projected Gradient Descent (PGD) is widely used to evaluate adversarial robustness, typically via final adversarial accuracy, which does not capture model behaviour throughout the attack.
By Dhairysheel Durgule
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
By Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt, Holger Rauhut
arXiv:2610. 23007v1 Announce Type: new Abstract: Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated.
By Quan Nguyen, Robin Yadav, Param Mody, Mark Schmidt
arXiv:2505. 02360v3 Announce Type: replace-cross Abstract: Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants.
By Fares B. Mehouachi, Saif Eddin Jabari
arXiv:2505. 01423v2 Announce Type: replace-cross Abstract: Efficient computation of min-max problems is a central question in optimization, learning, games, and control.
By Henry Shugart, Jason M. Altschuler
arXiv:2606. 00738v1 Announce Type: cross Abstract: Adversarial Training (AT) is a leading defense against adversarial examples but often suffers from Catastrophic Overfitting (CO) in efficient single-step variants, where robustness to multi-step attacks collapses despite high single-step performance.
By Mazdak Teymourian, Ramtin Moslemi, Farzan Rahmani, Mohammad Hossein Rohban
The paper introduces CoDRA, a cost-to-disturbance ratio approach for adversarial reinforcement learning that balances controller performance and disturbance exposure without extra penalty terms. CoDRA uses a self‑normalized actor–critic update, scaling value terms by a stop‑gradient normalization constant derived from the current batch. Experiments on MuJoCo pendulum tasks show that CoDRA achieves the lowest cost across a range of forces and masses, outperforming other methods especially on the more challenging InvertedDoublePendulum environment.
By Taeho Lee, Donghwan Lee
Controlled-Minimax is a new framework that stabilizes adversarial minimax optimization by modeling training dynamics as a second-order ODE and applying a feedback controller derived from control theory. The method guarantees local exponential stability regardless of the Hessian spectrum, addressing persistent limit cycles and divergence that plague traditional gradient descent-ascent approaches. Experiments on benchmarks such as Dirac-GAN, 2D Gaussian mixtures, Swiss Roll, and StyleGAN2-ada show that Controlled-Minimax suppresses oscillations and achieves fast, robust convergence.
By Tianjiao Luo, Jianfei Chen, Jun Zhu
Deep neural networks can exhibit periodic loss spikes known as the "Slingshot Mechanism" during long‑term training. This paper shows that such spikes arise from finite‑precision floating‑point arithmetic: when the correct‑class logit becomes much larger than the others, the gradient for that class is rounded to zero while others remain nonzero, breaking the zero‑sum constraint and creating a positive feedback loop that inflates feature and classifier norms. The authors term this Numerical Feature Inflation (NFI) and demonstrate that it explains the rapid norm growth, reappearance of gradients, and loss spikes, while also showing that partial absorption can still cause hidden norm growth without visible spikes.
By Hanqing Liu, Jianjun Cao, Yuanze Li, Zijian Zhou