arXiv Machine Learning
Jun 8
Flatland: The Adventures of Gradient Descent with Large Step Sizes
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
arXiv:2507. 20424v3 Announce Type: replace Abstract: We study centralized distributed data parallel training of deep neural networks (DNNs), aiming to improve the trade-off between communication efficiency and model performance of the local gradient methods.
Related stories
arXiv Machine Learning
Jun 16
Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learning
arXiv:2512. 12737v2 Announce Type: replace Abstract: Decentralized federated learning (DFL) enables collaborative model training without a central server, but converges slowly under statistical heterogeneity.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Aug 11
Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.
arXiv Machine Learning
1d ago
On the Escaping Efficiency of Distributed Adversarial Training Algorithms
The paper compares distributed adversarial training algorithms—both centralized and decentralized—within multi‑agent learning environments. It introduces a theoretical framework to analyze how efficiently these algorithms escape local minima, a property linked to model flatness and robustness. The study finds that with small perturbation bounds and large batch sizes, decentralized methods (consensus and diffusion) escape local minima faster than centralized ones, but this advantage may diminish as attack strength increases.
arXiv Machine Learning
Aug 11
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
arXiv:2502. 11152v4 Announce Type: replace-cross Abstract: The optimization foundations of deep linear networks have recently received significant attention.
arXiv AI
Aug 26
Across the Loss Landscape with Progressive Growth
arXiv:2608.24568v1 Announce Type: cross Abstract: Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geomet...
arXiv AI
Jun 2
Stability Analysis of Sharpness-Aware Minimization
arXiv:2301. 06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains.
arXiv Machine Learning
Aug 4
GradientStabilizer:Fix the Norm, Not the Gradient
arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.
arXiv Machine Learning
Aug 27
How Edge of Stability Hinders SCAFFOLD in Federated Optimization
The paper investigates why the SCAFFOLD algorithm, designed to be robust to data heterogeneity in federated learning, often underperforms compared to the simpler FedAvg. It identifies the presence of Edge of Stability (EoS) dynamics and progressive sharpening as key factors, showing that both algorithms exhibit EoS behavior across various architectures and hyperparameters. Crucially, the study finds that at the EoS, SCAFFOLD’s ability to estimate the global gradient deteriorates, as indicated by a weakened correlation between sharpness and gradient estimation error, explaining its limited practical advantage.
arXiv Machine Learning
Jul 8
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv Machine Learning
5d ago
AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery
AYLA is a loss reparameterization framework that applies a sigmoid‑controlled power‑law transformation to the empirical loss, dynamically adjusting gradient magnitudes without changing stationary points or optimal solutions. By reshaping optimization trajectories, AYLA accelerates descent in flat or saddle‑dominated regions and stabilizes late‑stage training, leading to improved feature recovery in two‑layer tanh networks on synthetic Gaussian data. Experiments show enhanced weight alignment, neuron similarity, activation correlation, and richer internal representations, while mitigating rank collapse and promoting a transition from lazy to active feature‑learning regimes.