← Back to all news
arXiv Machine Learning By Yi Feng, Andrea Paudice

Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Sharpness

Read the original on arXiv Machine Learning →

The paper investigates how deterministic Gradient Descent behaves when applied to Adversarial Training with large, constant step sizes. It analyzes both a linear logistic regression setting—examining where parameters converge (implicit bias) and how fast losses decrease (optimization)—and neural network training, focusing on the evolution of loss sharpness. The study reveals three key distinctions from Standard Training: the implicit bias gap persists even with large steps, there is no uniform convergence rate for optimization error, and sharpness follows a two‑stage dynamic unlike the Edge‑of‑Stability pattern seen in Standard Training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 12

On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

The paper analyzes Gradient Descent (GD) on a minimal binary classification problem with a two‑neuron ReLU network trained on two instances. It proves that GD converges to an optimal robustness margin—maximizing the distance between the decision boundary and training points—but does so at a very slow rate, specifically θ(1/ln(t)). Empirical simulations confirm this slow convergence across various initializations, and the authors provide a rigorous analysis of GD trajectories across different activation patterns to establish the first explicit lower bound on robustness convergence in a non‑linear model.

By Guy Smorodinsky, Sveta Gimpleson, Itay Safran
safety
More like this →