Fit More and Train Faster With ZeRO via DeepSpeed and FairScale

Accelerate Large Model Training using DeepSpeed

Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

arXiv:2603. 28921v3 Announce Type: replace-cross Abstract: The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)).

Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

The paper introduces Repeated Optimizer Resampling (ROR), a method that treats optimizer choice as a hyperparameter and searches for the best optimizer during a single training run. ROR periodically scouts each candidate optimizer for a short number of epochs, then continues training with the best scout, allowing the optimizer to change over time. Experiments on MNIST, Fashion‑MNIST, and motor insurance claim‑count models show that one‑epoch ROR uses only 24–35% of the training effort required to exhaustively evaluate all optimizers while achieving comparable performance.

M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

arXiv:2607. 10611v1 Announce Type: new Abstract: Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies.

Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

arXiv:2607. 14516v1 Announce Type: new Abstract: Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks.

Velocity Scheduled Flow Matching

Flow matching trains a neural network to regress the conditional velocity along a linear interpolant between noise and data, and the number of network evaluations~(NFE) sets the cost of sampling. The straight-line interpolant carries an implicit choice: the sample moves at constant speed throughout the trajectory.

Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

arXiv:2608. 02829v1 Announce Type: new Abstract: Model families train every size from scratch.

How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

arXiv:2607. 01487v1 Announce Type: new Abstract: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law).

From DeepSpeed to FSDP and Back Again with Hugging Face Accelerate

Speedrunning Tabular Foundation Model Pretraining

arXiv:2606. 03681v1 Announce Type: new Abstract: Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimization ideas.

From PyTorch DDP to Accelerate to Trainer, mastery of distributed training with ease

20x Faster TRL Fine-tuning with RapidFire AI