arXiv AI
FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 17
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.
arXiv AI
2d ago
FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection
FedLore introduces a communication- and memory-efficient federated learning framework that shares a low-rank optimization basis across clients each round, mitigating subspace fragmentation and enabling exact low-rank aggregation. By refreshing this shared basis across rounds, FedLore allows model updates to exceed the per-round rank budget while maintaining a provable $O(T^{-1/2})$ stationarity bound under standard assumptions. Experiments on vision and language tasks, including federated pre‑training, demonstrate that FedLore outperforms low‑rank adapter baselines and matches or surpasses full‑parameter training while reducing communication and optimizer‑state memory.
arXiv Machine Learning
Aug 5
Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.
arXiv Machine Learning
Sep 10
MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models
arXiv:2609.07666v1 Announce Type: new Abstract: Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order...
arXiv Machine Learning
Jul 28
Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided H\"older Regularity
arXiv:2607. 22906v1 Announce Type: new Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided H\"older regularity.
arXiv Machine Learning
Aug 11
Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum
arXiv:2504. 12742v2 Announce Type: replace Abstract: Decentralized Federated Learning (DFL) enables collaborative model training without relying on a central server.