The paper introduces Bilateral Sharpness-Aware Minimization (BSAM), a new optimization method that combines the traditional Max-Sharpness (MaxS) metric with a newly defined Min-Sharpness (MinS) metric. By using the difference between the training loss and the minimum loss in a neighborhood around the current weights, BSAM creates a more comprehensive Flatness Indicator (FI) that guides the search toward flatter minima. Experiments show that BSAM improves generalization and robustness over vanilla SAM in tasks such as classification, transfer learning, human pose estimation, and network quantization.
arXiv:2510.12451v3 Announce Type: replace-cross Abstract: Flat minima are strongly associated with improved generalisation in deep neural networks. However, this connection has proven nuanced in rece...
The paper investigates whether flatter loss landscapes drive better generalization in the context of Grokking. It shows that sharpness‑aware minimization (SAM) alone does not reliably induce the transition to generalization, but when combined with weight decay, SAM can accelerate this transition up to four times faster. The authors provide a theoretical analysis using a minimal two‑layer ReLU model, demonstrating that flatness alone cannot distinguish memorizing from generalizing solutions, while weight decay favors generalization and a local stability window explains SAM’s effect.
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv:2607. 18306v1 Announce Type: cross Abstract: Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss in a local parameter neighborhood.
arXiv:2409.09984v2 Announce Type: replace Abstract: The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generaliz...
arXiv:2608. 15105v1 Announce Type: new Abstract: Recent progress in optimization research has highlighted the sharpness of the loss landscape as a key factor in narrowing the generalization gap.
A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al. , 2017), where flatness can be measured by the trace of the Hessian of the empirical loss.
arXiv:2606. 04429v1 Announce Type: cross Abstract: A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al.
The paper investigates why saliency maps of adversarially‑trained neural networks appear sparse. It provides a theoretical explanation for two‑layer ReLU networks, showing that as data size and neuron count grow, minimizers converge to a Bayes classifier with minimal gradient and Barron norm. The sparsity arises because α−−‑training with λ−1 attacks yields anisotropic gradient norms that favor axis‑aligned, sparse gradients, a claim supported by experimental evaluation of gradient α‑norms and sparsity thresholds.
arXiv:2606. 20469v1 Announce Type: new Abstract: A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative.
The paper investigates why direct pairwise AUC surrogate training, though aligned with the ranking metric, can fail to provide corrective updates during stochastic training. It introduces the concept of Corrective Power, showing that in certain regions severe ranking errors persist without generating strong gradient signals, a phenomenon linked to degeneracies in the U‑statistics of pairwise gradients. The authors explain empirical observations—such as the effectiveness of squared‑hinge loss, the benefit of cross‑entropy warm‑up, and the advantage of parameter‑efficient fine‑tuning—through this theoretical lens, and validate their claims with experiments on eight computer‑vision and eight NLP tasks using various models.
We introduce Pre-Warm, a simple yet effective zero-training-cost method for data-conditioned initialization of the first convolutional layer. Before the first forward pass, Pre-Warm extracts mean-centered local patches from a single training batch, clusters them with MiniBatchKMeans, applies inverse Manhattan spatial weighting, and uses the resulting centroids to initialize half of the first-layer filters (the remainder retain Kaiming initialization).