The paper investigates how the timing of introducing mid‑training (MT) data and the decay of the learning rate (LR) interact in large‑language‑model training. It shows that when the LR decays, mixing MT data earlier with pretraining (PT) data reduces final MT loss, whereas with a constant LR the opposite holds. The authors conduct ablation studies and propose a theoretical model explaining why the optimal data schedule depends on the LR schedule.
By Haodong Wen, Kairong Luo, Tingkai Yan, Wenguang Chen, Kaifeng Lyu
The paper proposes a theoretical framework for scheduling high‑quality data in large language model training by extending functional scaling laws to account for time‑varying data quality. It identifies two regimes—noise‑limited and signal‑limited—where high‑quality data should be used differently, and introduces a Drop‑Stable‑Rampup training schedule that adjusts batch size at the quality transition. Experiments on 15B MoE and 600M dense models show significant accuracy gains over conventional decay schedules across multiple benchmarks.
By Zhitao Zhu, Xili Wang, Shizhe Wu, Jiawei Fu, Xiaoqing Liu
The paper introduces OP‑Mix, an on‑policy data mixing algorithm that operates continuously across pretraining, mid‑training, and instruction tuning. By simulating candidate mixtures through low‑rank adapters trained on the current model, OP‑Mix avoids separate proxy models and remains grounded in the model’s learning dynamics. Experiments show OP‑Mix consistently finds near‑optimal mixtures, improving pretraining perplexity by 6.3% and reducing compute for continual learning by 66–95% compared to baselines.
By Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho, Tal Linzen, Pratyusha Sharma
The paper studies how long‑context continual pre‑training (LCCP) scales with token budget and model size. An empirical additive power law is derived that predicts LCCP performance across five model scales and over a hundred budget configurations, and a new Constant‑Last4 protocol is introduced for stable monitoring. Experiments show that performance continues to improve without clear saturation, even after 128K context extension and fine‑tuning, indicating that scaling LCCP yields transferable gains for downstream tasks.
By Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao, Xu Han
arXiv:2607. 04969v1 Announce Type: new Abstract: The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency.
By Jingwei Zuo, Cong Zeng, Ilyas Chahed, Maksim Velikanov, Dhia Eddine Rhaiem, Pasquale Balsebre, Abhay Kumar, Younes Belkada, Hakim Hacid
Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this qu...
The paper investigates training-time data augmentation as a regularizer for autoregressive language model pretraining in data‑constrained, compute‑abundant settings. It introduces three orthogonal augmentation categories—token‑level noise, sequence permutations, and target offset prediction—and shows through systematic ablations that each category delays overfitting and reduces validation loss, with random token replacement performing best individually. Combining augmentation categories further lowers the minimum validation loss, demonstrating that such augmentations mitigate data inefficiency in autoregressive pretraining.
By Michael K. Chen, Xikun Zhang, Fan Bai, Zhengding Hu, Zhen Wang
The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.
By Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang
arXiv:2602.06797v3 Announce Type: replace-cross
Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...
By Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu
arXiv:2606. 03938v1 Announce Type: cross Abstract: Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text.
By Bishwas Mandal, Shmuel Berman, Akshay Vegesna, Samip Dahal
arXiv:2609.13922v1 Announce Type: new
Abstract: Minibatch persistency reuses data instead of reading it: rather than drawing a fresh minibatch at every optimizer step, it takes K consecutive steps on...
By Matteo Fischetti
arXiv:2602.03702v2 Announce Type: replace-cross
Abstract: Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance....
By Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade