Zephon is a data loader designed for foundation model training that ensures deterministic ordering of training data batches even when GPU resources change, checkpoints are resumed, or different processing backends are used. It handles online, stateful pipelines—where tokenization, packing, and mixing of samples create complex n‑to‑m transformations—by partitioning the data stream into topology‑independent lanes, serializing ordering decisions, and checkpointing only bounded in‑flight state. Experiments on text and vision‑language tasks show Zephon delivers competitive throughput while offering guarantees that existing loaders lack for such pipelines.
By Maximilian B\"other, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
NeMo-DCR is a delta‑compressed refit method that sends only the changes in a trillion‑parameter policy model, achieving bit‑exact synchronization across AWS regions. By using fixed affine mappings, XOR masks, and a relay‑tree object storage, it avoids full checkpoint transfers and cross‑cluster collectives, reducing refit times from 87.5 min to 150 s for a 1 T model at 3% change rate. The approach scales to 30 B–1 T models, delivering 12–40× speedups over traditional full‑checkpoint refits.
By Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco
arXiv:2605.18904v2 Announce Type: replace-cross
Abstract: Model merging consolidates independently fine-tuned checkpoints without joint training or access to task data. Dynamic methods retain task-sp...
By Guodong Du, Wanyu Lin
Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight...
arXiv:2610.26179v2 Announce Type: replace
Abstract: Hierarchical world models plan over long horizons by passing predicted states from a coarse model to a lower-level controller. Choosing these state...
By Xiang Kaibo, Liu Qingfeng
arXiv:2609.37169v1 Announce Type: cross
Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
By Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou