arXiv Machine Learning June 25, 2026 By Kwok Chun Au, Adam Block

Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

Read the original on arXiv Machine Learning →

arXiv:2606. 25086v1 Announce Type: new Abstract: Many modern Language Model (LM) pipelines return an averaged model, such as an exponential moving average of the training iterates, rather than the final iterate itself.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 26

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

arXiv:2602.03702v2 Announce Type: replace-cross Abstract: Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance....

By Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade

llms

arXiv AI

Jul 28

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling

arXiv:2510. 14717v2 Announce Type: replace-cross Abstract: Increasing the batch size during training -- a ''batch ramp'' -- is a promising strategy to accelerate large language model pretraining.

By Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, Sham Kakade

llms

arXiv Machine Learning

Jun 16

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

arXiv:2603. 09923v4 Announce Type: replace Abstract: Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam.

By Ganzhao Yuan

arXiv AI

Jun 3

q0: Primitives for Hyper-Epoch Pretraining

arXiv:2606. 03938v1 Announce Type: cross Abstract: Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text.

By Bishwas Mandal, Shmuel Berman, Akshay Vegesna, Samip Dahal

efficiency benchmarks

arXiv Machine Learning

4d ago

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

arXiv:2609.36945v1 Announce Type: new Abstract: We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version...

By Zhiwei Wang, Yanxi Chen, Yaliang Li, Bolin Ding

llms reinforcement-learning fine-tuning

arXiv AI

Sep 1

When Do Larger Batches Help Scale LLM Reinforcement Learning?

arXiv:2608.29296v1 Announce Type: cross Abstract: Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this stati...

By Ziniu Li, Jinbo Wang, Guanhua Huang, Feiyuan Zhang, Pengbo Li, Alex Chen

llms reinforcement-learning efficiency

One email a morning, machine-written

Related stories

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling

OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

q0: Primitives for Hyper-Epoch Pretraining

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

When Do Larger Batches Help Scale LLM Reinforcement Learning?