Understanding Practical Optimization in Machine Learning
Read the original on arXiv Machine Learning →The thesis investigates practical optimization techniques used in modern machine learning, such as submodel training, gradient clipping, and layer-wise non‑Euclidean directions, by analyzing their actual objectives, assumptions, updates, and computations rather than simplified proxies. It demonstrates that Independent Subnetwork Training can converge to a shifted fixed point due to sketching bias, while Model‑Agnostic Sparsified Training incorporates the sketch distribution to achieve unbiased gradients and convergence guarantees. The work also explores private optimization, showing that adaptive quantile clipping retains bias unless a suitable schedule is used, and combines smoothed normalization with error compensation to obtain distributed private convergence without bounded gradient assumptions. Additionally, it develops a layer‑wise generalized smoothness model for optimizers based on linear minimization oracles, validates it through the Gluon framework, and highlights the importance of precise orthogonalization in Muon‑style methods.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.