Fit More and Train Faster With ZeRO via DeepSpeed and FairScale

Accelerate Large Model Training using DeepSpeed

Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

arXiv:2603. 28921v3 Announce Type: replace-cross Abstract: The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)).

LESS: Lightweight Evolutionary Supernet Search in Minutes

LESS (Lightweight Evolutionary Supernet Search) is a data‑driven NAS method that uses a brief hard‑path warm‑up and CMA‑ES to evaluate candidate architectures as decoded hard genotypes after six supernet updates. On NAS‑Bench‑201, LESS attains 93.189 % CIFAR‑10 accuracy in just 409.1 seconds, nearly matching FairNAS while using only about 1/24 of its search time. The approach also transfers well to CIFAR‑100, ImageNet16‑120, and the larger DARTS space, achieving high accuracies with searches completed in roughly 43.5 minutes on a single GPU.

Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M

arXiv:2609.14715v1 Announce Type: new Abstract: We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query...

Miles v0.1: Production-Level Post-Training

arXiv:2609.08368v1 Announce Type: new Abstract: We present Miles v0.1, a full-stack, production-ready system for frontier post-training. Building upon the clean design of slime, Miles designs each st...

Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

The paper introduces Repeated Optimizer Resampling (ROR), a method that treats optimizer choice as a hyperparameter and searches for the best optimizer during a single training run. ROR periodically scouts each candidate optimizer for a short number of epochs, then continues training with the best scout, allowing the optimizer to change over time. Experiments on MNIST, Fashion‑MNIST, and motor insurance claim‑count models show that one‑epoch ROR uses only 24–35% of the training effort required to exhaustively evaluate all optimizers while achieving comparable performance.