Accelerate Large Model Training using DeepSpeed

Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel

Training Variable Long Sequences with Data-Centric Parallel

arXiv:2608. 07524v1 Announce Type: new Abstract: Training deep learning models on variable long sequences poses significant computational challenges.

GaLore: Advancing Large Model Training on Consumer-grade Hardware

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.

Efficient Scaling of LLM Training with Flexible Context Parallelism

arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).

Scaling depth capacity via zero/one-layer model expansion

arXiv:2511. 04981v2 Announce Type: replace Abstract: Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost.

Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks

Deep neural networks have witnessed remarkable advancements in recent years and have become integral to various applications. However, alongside these developments, training and deployment of neural network models on embedding and edge devices face significant challenges due to limited memory and computational resources.

$\mu$pscaling small models: Principled warm starts and hyperparameter transfer

arXiv:2602. 10545v2 Announce Type: replace-cross Abstract: Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.

MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

arXiv:2606. 17526v1 Announce Type: new Abstract: Efficient optimization is essential for training large language models.

Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models

arXiv:2607. 01844v1 Announce Type: cross Abstract: This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.

DMuon: Efficient Distributed Muon Training with Near-Adam Overhead

arXiv:2606. 27153v1 Announce Type: cross Abstract: Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads.

Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism

arXiv:2506. 01260v2 Announce Type: replace Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks.