The paper introduces Sparsity-Adaptive Sharpness-Aware Minimization (SA‑SAM), a method that adjusts the perturbation radius in sharpness-aware training to remain consistent as model sparsity increases. It also evaluates a Magnitude‑Weighted Hessian (MWH) importance metric derived from second‑order analysis. Experiments on CIFAR‑10‑C, CIFAR‑100‑C, and ImageNet‑100‑C show that SA‑SAM improves corruption robustness at 80–90% sparsity while maintaining clean accuracy, and the study reports inference throughput at deployment‑relevant sparsity levels.
arXiv:2608. 06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs.
arXiv:2506.17874v3 Announce Type: replace-cross Abstract: In many real-world applications, ensuring the robustness and stability of deep neural networks (DNNs) is crucial, particularly for image clas...
arXiv:2607. 14466v1 Announce Type: new Abstract: Noise injection is a well-known technique in stochastic optimization.
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv:2610.32465v1 Announce Type: new Abstract: Adversarial sharpness penalty per iteration of a base optimizer, known as sharpness-aware minimization (SAM), is an accepted technique to improve gener...
arXiv:2604.13287v2 Announce Type: replace Abstract: Weight pruning is a common technique for compressing large neural networks. We focus on the challenging post-training one-shot setting, where a pre...
arXiv:2606. 02267v1 Announce Type: new Abstract: The vulnerability of deep neural networks to adversarial examples poses a significant challenge for real-world deployment.
arXiv:2610.20415v1 Announce Type: new Abstract: Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima. Sharpness-Aware Minimisation (SAM) improves gene...
arXiv:2608. 15105v1 Announce Type: new Abstract: Recent progress in optimization research has highlighted the sharpness of the loss landscape as a key factor in narrowing the generalization gap.
The paper introduces Fast Preemptive Robustification (FPR), a lightweight defense that enhances the robustness of deep neural networks against transferable adversarial attacks. By sharpening Laplacian responses through a single 3×3 channel‑wise convolution, FPR eliminates the need for surrogate models, iterative optimization, or specialized training. Experiments show that FPR lowers untargeted attack success rates by 12.7% and reduces targeted attack success from 10.7% to 4.1%.
Sparse autoencoders (SAEs) are commonly used to interpret large language models, but their reliability after pruning is unclear. This study shows that pruning’s effect on an SAE is governed by perturbation energy, a covariance-weighted norm, and that magnitude pruning distorts the representation space by ignoring activation geometry. Activation-aware pruning methods such as Wanda and SparseGPT better preserve SAE behavior, and the authors find that middle layers are especially vulnerable, leading them to propose a layer‑wise sparsity allocation that reduces perplexity for a given sparsity level.
The paper introduces Curvature-Scaled Sharpness-Aware Minimization (CS‑SAM), a variant of SAM that scales both the perturbation and descent step by curvature bounds to uniformly control gradient distortion under generalized smoothness conditions. It proves deterministic ≥1/T convergence rates for nonconvex objectives, last‑iterate ≥1/T rates for convex objectives, and linear convergence under the Polyak‑Łojasiewicz condition, while recovering standard gradient descent rates when the perturbation is zero. Experiments on computer vision tasks show that CS‑SAM, particularly the (H0,H1) variant, achieves the highest mean best‑test accuracy across four settings compared to tuned SAM and USAM.