arXiv AI
Aug 11
Training Variable Long Sequences with Data-Centric Parallel
arXiv:2608. 07524v1 Announce Type: new Abstract: Training deep learning models on variable long sequences poses significant computational challenges.
Pipelined Gradient Coding
arXiv:2607. 20739v1 Announce Type: cross Abstract: In large-scale machine learning, distributed training commonly involves multiple workers evaluating the gradients of the model on different dataset partitions.
Related stories
arXiv AI
Jul 28
cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs
arXiv:2607. 22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes.
arXiv AI
Jun 2
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
arXiv:2606. 01155v1 Announce Type: cross Abstract: Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not.
arXiv Machine Learning
2d ago
MLCC: A Congestion Control Technique to Accelerate ML Training
arXiv:2402. 09589v2 Announce Type: replace-cross Abstract: We present MLCC, a novel technique to augment today's congestion control algorithms to accelerate DNN training jobs in shared GPU clusters in a fully distributed manner.
arXiv AI
Jul 2
scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
arXiv:2506. 01883v3 Announce Type: replace-cross Abstract: Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory.
arXiv AI
Jun 10
Piper: A Programmable Distributed Training System
arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.
arXiv AI
Jul 7
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
arXiv:2509. 23722v2 Announce Type: replace-cross Abstract: Pipeline parallelism is widely used to train large language models (LLMs).
arXiv AI
1d ago
VirnyFlow: Optimizing ML Pipelines for Accuracy, Fairness, and Stability at Scale
arXiv:2506. 01584v2 Announce Type: replace-cross Abstract: Developing machine learning (ML) systems for real-world deployment requires navigating context-dependent trade-offs among accuracy, fairness, stability, and other objectives.
arXiv Machine Learning
Jun 9
Efficient Scaling of LLM Training with Flexible Context Parallelism
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
arXiv Machine Learning
Aug 10
Optimization as a Dynamical System: Generative Schedules from Latent ODEs
arXiv:2509. 23052v2 Announce Type: replace Abstract: We present a new meta-learning method to determine the optimal learning rate schedule for gradient descent.
arXiv Machine Learning
Jul 9
Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism
arXiv:2506. 01260v2 Announce Type: replace Abstract: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks.
arXiv Machine Learning
Jul 16
GFlowRL: Scaling Distribution-Matching RL to Large Language Models
arXiv:2607. 13394v1 Announce Type: cross Abstract: Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes.