arXiv AI
SoftServe: A Scalable Quasi-Newton Method for Deep Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 2
Exploiting weight-space symmetries for approximating curvature
arXiv:2606. 00442v1 Announce Type: new Abstract: Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks.
arXiv Machine Learning
Aug 4
Simplified Quadratic Gradient: A Unified Framework Bridging Gradient Descent and Newton-Type Methods by Synthesizing Hessians and Gradients
arXiv:2209. 03282v5 Announce Type: replace-cross Abstract: Accelerating the convergence of second-order optimization, particularly Newton-type methods, remains a pivotal challenge in algorithmic research.
arXiv Machine Learning
Jun 9
LEAF: A Learning-Enabled ADMM Framework for Accelerated Convex Optimization
arXiv:2606. 08993v1 Announce Type: new Abstract: We propose LEAF, a learning-enabled ADMM framework for accelerated convex optimization.
arXiv Machine Learning
Aug 11
Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.
arXiv Machine Learning
Jul 7
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
arXiv:2406. 14340v2 Announce Type: replace-cross Abstract: The standard stochastic gradient descent (SGD) optimization method, as well as adaptive methods such as the Adam optimizer fail to converge if the learning rates do not converge to zero (particularly, in the situation of constant learning rates).
arXiv AI
Jun 2
Stability Analysis of Sharpness-Aware Minimization
arXiv:2301. 06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains.