arXiv:2607. 25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data.
By Tian Qin, Kimia Hamidieh, David Alvarez-Melis
arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.
By Zhiwei Xu, Shihao Wu, Hanseul Cho, Wei Hu, Yixin Wang
arXiv:2606. 01155v1 Announce Type: cross Abstract: Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not.
By Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, Decebal Constantin Mocanu
arXiv:2608. 14071v1 Announce Type: new Abstract: As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)).
By Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang
arXiv:2606. 16246v1 Announce Type: cross Abstract: As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora.
By Michael K. Chen, Xikun Zhang, Zhen Wang
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
By Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade