Mid-training (MT) is a training phase in LLM training pipelines that continues pretraining (PT) on smaller datasets of higher value. Taking the PT and MT phases as a whole, training usually starts wit...
The paper proposes a theoretical framework for scheduling high‑quality data in large language model training by extending functional scaling laws to account for time‑varying data quality. It identifies two regimes—noise‑limited and signal‑limited—where high‑quality data should be used differently, and introduces a Drop‑Stable‑Rampup training schedule that adjusts batch size at the quality transition. Experiments on 15B MoE and 600M dense models show significant accuracy gains over conventional decay schedules across multiple benchmarks.
By Zhitao Zhu, Xili Wang, Shizhe Wu, Jiawei Fu, Xiaoqing Liu
arXiv:2609.13922v1 Announce Type: new
Abstract: Minibatch persistency reuses data instead of reading it: rather than drawing a fresh minibatch at every optimizer step, it takes K consecutive steps on...
By Matteo Fischetti
The paper introduces OP‑Mix, an on‑policy data mixing algorithm that operates continuously across pretraining, mid‑training, and instruction tuning. By simulating candidate mixtures through low‑rank adapters trained on the current model, OP‑Mix avoids separate proxy models and remains grounded in the model’s learning dynamics. Experiments show OP‑Mix consistently finds near‑optimal mixtures, improving pretraining perplexity by 6.3% and reducing compute for continual learning by 66–95% compared to baselines.
By Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma
The paper investigates training-time data augmentation as a regularizer for autoregressive language model pretraining in data‑constrained, compute‑abundant settings. It introduces three orthogonal augmentation categories—token‑level noise, sequence permutations, and target offset prediction—and shows through systematic ablations that each category delays overfitting and reduces validation loss, with random token replacement performing best individually. Combining augmentation categories further lowers the minimum validation loss, demonstrating that such augmentations mitigate data inefficiency in autoregressive pretraining.
By Michael K. Chen, Xikun Zhang, Fan Bai, Zhengding Hu, Zhen Wang
The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.
By Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang