The paper investigates why the SCAFFOLD algorithm, designed to be robust to data heterogeneity in federated learning, often underperforms compared to the simpler FedAvg. It identifies the presence of Edge of Stability (EoS) dynamics and progressive sharpening as key factors, showing that both algorithms exhibit EoS behavior across various architectures and hyperparameters. Crucially, the study finds that at the EoS, SCAFFOLD’s ability to estimate the global gradient deteriorates, as indicated by a weakened correlation between sharpness and gradient estimation error, explaining its limited practical advantage.
By Anant Khandelwal, Michael Crawshaw, Mingrui Liu
arXiv:2301. 06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains.
By Hoki Kim, Jinseong Park, Yujin Choi, Jaewook Lee
arXiv:2606. 04212v1 Announce Type: new Abstract: Existing analyses of the edge of stability (EoS) treat it as a global property of optimization.
By Shauna Kwag, Anakha Ganesh, Tomaso Poggio, Pierfrancesco Beneventano
arXiv:2605. 29547v2 Announce Type: replace-cross Abstract: Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators.
By Ruoran Xu, Borong She, Xiaobo Jin, Qiufeng Wang
arXiv:2606. 18080v1 Announce Type: new Abstract: Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold $2/\eta$, where $\eta$ is the learning rate.
By Pierre Marion
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
By Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang
arXiv:2510. 03164v2 Announce Type: replace Abstract: Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood.
By Foivos Alimisis, Rustem Islamov, Aurelien Lucchi
Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $ρ$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness.
arXiv:2608. 03197v1 Announce Type: new Abstract: Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear.
By Jiaxin Deng, Junbiao Pang
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
By Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus, Mark Schmidt, Holger Rauhut
arXiv:2608.24568v1 Announce Type: cross
Abstract: Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geomet...
By Paul Caillon, Christophe Cerisara, Alexandre Allauzen
The paper investigates the "edge of stability" phenomenon in deep learning, where Hessian eigenvalues remain stable above a classically predicted unstable threshold. It shows that many first‑order optimizers, including gradient descent, can violate this stability bound by up to a factor of 21.1, and that this deviation depends systematically on the optimizer used. The authors propose a new stability threshold based on the directional Hessian and gradient‑alignment score, which removes optimizer‑dependent offsets and offers consistent predictions while providing diagnostic tools to understand how optimizers balance temporal and spatial budgets.
By Jaerin Lee, Kyoung Mu Lee