arXiv Machine Learning August 31, 2026 By Niccol\`o Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, J\"org Franke, Sampo Pyysalo, Jenia Jitsev, Aaron Klein

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

Read the original on arXiv Machine Learning →

The paper investigates how learning rate and batch size scale when pretraining dense large language models on English‑prevalent corpora, examining both jointly optimal and marginal evolutions across model capacity and data size. It explores the benefits of a Warmup‑Stable‑Decay learning‑rate schedule, assessing whether optimal hyperparameters transfer between stable and decay phases, and evaluates loss scaling forms that capture interactions between model capacity and dataset size. The study provides a baseline scaling procedure and releases the full set of pretraining runs for future OpenEuroLLM development.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv Machine Learning

Jul 31

Towards joint scaling laws with optimal batch size schedules

arXiv:2607. 27731v1 Announce Type: new Abstract: Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics.

By Jiaxiang Li, Zhiqi Bu, Shiyun Xu

llms

arXiv AI

Jun 30

On the Nonlinearity of Learning Rate Scaling for LLM Training

arXiv:2606. 29158v1 Announce Type: cross Abstract: Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs.

By Zaiwen Yang, Huaqing Zhang, Jing Xu, Jingzhao Zhang

llms

arXiv AI

Aug 26

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

arXiv:2602.03702v2 Announce Type: replace-cross Abstract: Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance....

By Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade

llms

arXiv Machine Learning

Jun 8

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

arXiv:2606. 06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus.

By Zhiwei Xu, Shihao Wu, Hanseul Cho, Wei Hu, Yixin Wang

llms diffusion

Hugging Face Trending Papers

Jul 28

Bridging Compute- and Data-Optimal Pretraining

Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound.

arXiv Machine Learning

Jun 8

Inferring the Size of Large Language Models From Popular Text Memorization

arXiv:2605. 29223v3 Announce Type: replace Abstract: The parameter counts of the most widely used large language models (LLMs) are often withheld by their developers, leaving model size -- a primary reference point for interpreting capabilities and costs -- largely undisclosed.

By Ivica Nikolic

llms

One email a morning, machine-written

Related stories

Towards joint scaling laws with optimal batch size schedules

On the Nonlinearity of Learning Rate Scaling for LLM Training

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

Bridging Compute- and Data-Optimal Pretraining

Inferring the Size of Large Language Models From Popular Text Memorization