Zephon is a data loader designed for foundation model training that ensures deterministic ordering of training data batches even when GPU resources change, checkpoints are resumed, or different processing backends are used. It handles online, stateful pipelines—where tokenization, packing, and mixing of samples create complex n‑to‑m transformations—by partitioning the data stream into topology‑independent lanes, serializing ordering decisions, and checkpointing only bounded in‑flight state. Experiments on text and vision‑language tasks show Zephon delivers competitive throughput while offering guarantees that existing loaders lack for such pipelines.
By Maximilian B\"other, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
NeMo-DCR is a delta‑compressed refit method that sends only the changes in a trillion‑parameter policy model, achieving bit‑exact synchronization across AWS regions. By using fixed affine mappings, XOR masks, and a relay‑tree object storage, it avoids full checkpoint transfers and cross‑cluster collectives, reducing refit times from 87.5 min to 150 s for a 1 T model at 3% change rate. The approach scales to 30 B–1 T models, delivering 12–40× speedups over traditional full‑checkpoint refits.
By Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco
arXiv:2605.18904v2 Announce Type: replace-cross
Abstract: Model merging consolidates independently fine-tuned checkpoints without joint training or access to task data. Dynamic methods retain task-sp...
By Guodong Du, Wanyu Lin
Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight...
arXiv:2610.26179v2 Announce Type: replace
Abstract: Hierarchical world models plan over long horizons by passing predicted states from a coarse model to a lower-level controller. Choosing these state...
By Xiang Kaibo, Liu Qingfeng
arXiv:2609.37169v1 Announce Type: cross
Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
By Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou
Merge++ is a post‑hoc refinement technique for model merging that synthesizes task‑representative images by inverting expert checkpoints and then distills expert knowledge into a merged model. It operates without any additional data beyond the checkpoints and can be applied universally across existing weight‑space merging algorithms. Experiments show consistent improvements, with average gains of +2 to +8 points and up to +25.9 on specific configurations.
By Aditya Pola, Vineeth N. Balasubramanian
PROOF-Gen is a method that improves distillation of tool‑calling models by recovering successful trajectories from teacher failures. It uses per‑scenario prompt optimization to generate corrective guidance that steers the teacher to a passing trajectory, then removes this guidance before training so the student learns from clean demonstrations. On τ2‑bench, PROOF-Gen recovers 93% of failed scenarios, boosting Qwen3‑4B‑Instruct‑2507’s Pass^1 from 0.132 to 0.529 and improving Gemma 4 E4B‑it by 7.2pp on BFCL v4 multi‑turn, while also raising deployed on‑device model performance by up to 5.0pp across response‑quality metrics.
By Anh Ta, Junjie Zhu, Shahin Shayandeh
arXiv:2608. 14650v1 Announce Type: new Abstract: Existing adaptive-inference and world-action-model systems use cheap-stage outputs or predicted futures to allocate additional computation.
By Malo de Pastor
The study evaluates self‑supervised pretraining for Prognostics and Health Management (PHM) across 47 public datasets, using a compact Transformer encoder trained with masked patch prediction. Transfer performance is highly task‑dependent: it dramatically improves fault‑classification on CWRU and HSG18 but offers no reliable benefit for remaining‑useful‑life estimation on CMAPSS. The authors also compare centralized versus simulated federated pretraining, finding that centralized training generally outperforms federated training under matched budgets, and that the intrinsic SSL loss does not reliably predict downstream performance.
By Jos\'e Sanz-Dur\'an, David Sol\'is-Mart\'in
The paper introduces OrbitTrace, a benchmark of 50 physics‑grounded compute‑availability traces from satellite orbits, and investigates whether specialized interruption‑resilient optimizers are needed when training is interrupted by predictable compute gaps. Experiments on CIFAR‑10/ResNet‑18 and GPT‑2/AdamW show that a strong checkpoint‑and‑resume baseline that preserves full optimizer state and indexes learning‑rate schedules in effective time matches uninterrupted training, rendering most availability‑aware methods unnecessary. Only in a narrow regime—large models with non‑persistable optimizer state and frequent short pauses—does reactive adaptation recover a modest portion of the state‑loss penalty, and even this benefit disappears for eclipse‑scale gaps.
By Subhadip Mitra
arXiv:2608. 15089v1 Announce Type: new Abstract: Long-horizon agents can fail even when their underlying models can solve the constituent steps.
By Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang