arXiv:2606. 08578v1 Announce Type: new Abstract: Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models.
By Xu Zhang, Peang Wang, Wei Wang
The paper investigates how learning rate and batch size scale when pretraining dense large language models on English‑prevalent corpora, examining both jointly optimal and marginal evolutions across model capacity and data size. It explores the benefits of a Warmup‑Stable‑Decay learning‑rate schedule, assessing whether optimal hyperparameters transfer between stable and decay phases, and evaluates loss scaling forms that capture interactions between model capacity and dataset size. The study provides a baseline scaling procedure and releases the full set of pretraining runs for future OpenEuroLLM development.
By Niccol\`o Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, J\"org Franke, Sampo Pyysalo, Jenia Jitsev, Aaron Klein
arXiv:2510. 14717v2 Announce Type: replace-cross Abstract: Increasing the batch size during training -- a ''batch ramp'' -- is a promising strategy to accelerate large language model pretraining.
By Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, Sham Kakade
The paper investigates how momentum influences optimization in a river‑valley loss landscape, where a low‑loss manifold is surrounded by steep orthogonal directions. It shows that momentum stabilizes large learning rates that vanilla gradient descent cannot tolerate, enabling faster progress along the river. The study also finds that in very flat, slowly spinning rivers, momentum itself does not directly accelerate tracking, but the larger permissible learning rate does.
By Miao Lu, Zeyu Bian, Kaiyue Wen, Beining Wu, Siyu Chen, Tianhao Wang, Zhiyuan Li
arXiv:2608.24568v1 Announce Type: cross
Abstract: Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geomet...
By Paul Caillon, Christophe Cerisara, Alexandre Allauzen
arXiv:2607. 27731v1 Announce Type: new Abstract: Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics.
By Jiaxiang Li, Zhiqi Bu, Shiyun Xu