arXiv Machine Learning
Jun 8
Flatland: The Adventures of Gradient Descent with Large Step Sizes
arXiv:2606. 06722v1 Announce Type: new Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth.
Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
arXiv:2507. 20424v3 Announce Type: replace Abstract: We study centralized distributed data parallel training of deep neural networks (DNNs), aiming to improve the trade-off between communication efficiency and model performance of the local gradient methods.
Related stories
arXiv Machine Learning
Jun 16
Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learning
arXiv:2512. 12737v2 Announce Type: replace Abstract: Decentralized federated learning (DFL) enables collaborative model training without a central server, but converges slowly under statistical heterogeneity.
arXiv Machine Learning
Jun 4
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv Machine Learning
Aug 11
Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.
arXiv Machine Learning
Aug 11
Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks
arXiv:2502. 11152v4 Announce Type: replace-cross Abstract: The optimization foundations of deep linear networks have recently received significant attention.
arXiv AI
Jun 2
Stability Analysis of Sharpness-Aware Minimization
arXiv:2301. 06308v2 Announce Type: replace-cross Abstract: Sharpness-aware minimization (SAM) is a training method that seeks to find flat minima in deep learning, resulting in state-of-the-art performance across various domains.
arXiv Machine Learning
Aug 4
GradientStabilizer:Fix the Norm, Not the Gradient
arXiv:2502. 17055v5 Announce Type: replace Abstract: Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence.
arXiv Machine Learning
Jul 8
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
arXiv:2607. 06151v1 Announce Type: new Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data.
arXiv Machine Learning
Aug 10
Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization
arXiv:2608. 06563v1 Announce Type: new Abstract: Machine learning and optimization have advanced together, with practical demands motivating new theory and theoretical breakthroughs enabling new applications.
arXiv Machine Learning
Jun 30
Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization
arXiv:2505. 22578v2 Announce Type: replace Abstract: The optimization of neural networks under weight decay remains poorly understood from a theoretical standpoint.
arXiv AI
2d ago
Decentralized Federated Learning for Heterogeneous Multi-Task Semantic Communication
arXiv:2608. 15256v1 Announce Type: new Abstract: Collaborative training in distributed semantic communication (DSC) networks typically relies on decentralized federated learning (DFL).
arXiv AI
Jul 15
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
arXiv:2607. 12112v1 Announce Type: cross Abstract: Federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks enables privacy-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross-modal representations.