← Back to all news
arXiv Machine Learning By Tao Huang, Chen Zhu, Antonio Alonso, Daming Li, Adarsh Goura, Jaideep Ray, Yitong Zhou

Coordination-Free Workload Regularization for Variable-Cost Distributed Training

Read the original on arXiv Machine Learning →

The paper introduces Batch Cost Lookahead (BCL), a coordination‑free data‑loading strategy that buffers a bounded window of records per rank and repartitions them into fixed‑size packs with balanced estimated cost. BCL maintains the trainer’s record‑count contract without requiring cross‑rank communication or runtime changes, and in simulations it reduces per‑rank batch‑cost variability by up to 82.8%. Experiments on 13 NeMo‑RL SFT corpora across eight H200 GPUs show speedups ranging from 1.050× to 1.167× over pooled MFFD, with further gains up to 1.343× on multi‑node setups, while also identifying a memory‑feasibility boundary that limits outliers before balancing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 30

Scaling Zero-Order Pretraining through Model Sharding

arXiv:2609.37899v1 Announce Type: new Abstract: Zero-order optimization (ZO) trains without backpropagation, making it relevant to forward-only hardware and non-differentiable loss, but its gradient...

By Francois Chaubard, Mykel J. Kochenderfer, Chris R\'e
More like this →
arXiv AI
Oct 16

Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe

The paper introduces an open midtraining recipe that primes base models for agentic post‑training, filling a gap between pretraining and full fine‑tuning. It trains on 256K context with hundreds of billions of tokens drawn from natural software‑engineering data and a small share of agentic traces. Applying this midtraining to Nemotron 3.5 Lightning and Super models yields measurable gains on SWE‑bench and DeepSWE benchmarks after identical full SFT, and improves robustness across agent harnesses.

By Soumye Singhal, Joseph Jennings, Ka Shun Shum, Alexander Bukharin, Khushi Bhardwaj, Zirui Liu, Venkat Srinivasan, Jonathan Li, Yash Jain, Kezhi Kong, Markus Kliegl, Olivier Delalleau, Sanjeev Satheesh, Mostofa Patwary
agentsreinforcement-learningfine-tuningbenchmarks
More like this →
arXiv AI
Oct 5

Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines

Zephon is a data loader designed for foundation model training that ensures deterministic ordering of training data batches even when GPU resources change, checkpoints are resumed, or different processing backends are used. It handles online, stateful pipelines—where tokenization, packing, and mixing of samples create complex n‑to‑m transformations—by partitioning the data stream into topology‑independent lanes, serializing ordering decisions, and checkpointing only bounded in‑flight state. Experiments on text and vision‑language tasks show Zephon delivers competitive throughput while offering guarantees that existing loaders lack for such pipelines.

By Maximilian B\"other, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
efficiencymultimodal
More like this →
arXiv Machine Learning
Jun 4

UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

arXiv:2606. 04101v1 Announce Type: cross Abstract: Large-scale expert parallelism (EP) is becoming pivotal for training and serving frontier MoE models, but it also amplifies device-level expert load imbalance into compute stragglers, token all-to-all bottlenecks, and activation-memory spikes.

By Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, Zhouyang Li, Yuliang Liu, Guojie Luo
efficiency
More like this →