Hugging Face Trending Papers
Understanding Practical Optimization in Machine Learning
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
5d ago
Understanding Practical Optimization in Machine Learning
The thesis investigates practical optimization techniques used in modern machine learning, such as submodel training, gradient clipping, and layer-wise non‑Euclidean directions, by analyzing their actual objectives, assumptions, updates, and computations rather than simplified proxies. It demonstrates that Independent Subnetwork Training can converge to a shifted fixed point due to sketching bias, while Model‑Agnostic Sparsified Training incorporates the sketch distribution to achieve unbiased gradients and convergence guarantees. The work also explores private optimization, showing that adaptive quantile clipping retains bias unless a suitable schedule is used, and combines smoothed normalization with error compensation to obtain distributed private convergence without bounded gradient assumptions. Additionally, it develops a layer‑wise generalized smoothness model for optimizers based on linear minimization oracles, validates it through the Gluon framework, and highlights the importance of precise orthogonalization in Muon‑style methods.
arXiv Machine Learning
6d ago
Sharpness-aware Orthogonalized Momentum Algorithms
arXiv:2610.32465v1 Announce Type: new Abstract: Adversarial sharpness penalty per iteration of a base optimizer, known as sharpness-aware minimization (SAM), is an accepted technique to improve gener...
arXiv Machine Learning
Aug 11
Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.
arXiv Machine Learning
Jun 5
Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations
arXiv:2606. 05599v1 Announce Type: new Abstract: This paper establishes a theoretical framework for the uniform convergence of smoothly activated deep neural network (DNN) estimators.
arXiv Machine Learning
Oct 21
Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training
arXiv:2608.09523v2 Announce Type: replace Abstract: Stochastic gradient descent (SGD) and its variants train deep neural networks (DNNs) effectively in practice, yet classical optimization theory doe...
arXiv Machine Learning
Oct 19
Towards Hyperparameter Transfer for Differentially Private Optimization
The paper introduces DP‑µP, a framework that extends Maximal Update Parametrization to differential privacy settings. By analyzing gradient clipping, noise injection, and feature learning, it derives a signal‑to‑noise ratio condition and spectral scaling rules for DP‑SGD, DP‑Adam, and DP‑Muon optimizers. Experiments show that DP‑µP enables zero‑shot transfer of learning rates across model widths on MLP, ViT, and GPT‑2, matching or surpassing tuned baselines while cutting computational costs.