Hugging Face Trending Papers
Data Schedules and Learning Rate Schedules Should Be Designed Together
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Oct 22
Data Schedules and Learning Rate Schedules Should Be Designed Together
The paper investigates how the timing of introducing mid‑training (MT) data and the decay of the learning rate (LR) interact in large‑language‑model training. It shows that when the LR decays, mixing MT data earlier with pretraining (PT) data reduces final MT loss, whereas with a constant LR the opposite holds. The authors conduct ablation studies and propose a theoretical model explaining why the optimal data schedule depends on the LR schedule.
arXiv AI
Aug 25
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
The paper proposes a theoretical framework for scheduling high‑quality data in large language model training by extending functional scaling laws to account for time‑varying data quality. It identifies two regimes—noise‑limited and signal‑limited—where high‑quality data should be used differently, and introduces a Drop‑Stable‑Rampup training schedule that adjusts batch size at the quality transition. Experiments on 15B MoE and 600M dense models show significant accuracy gains over conventional decay schedules across multiple benchmarks.
arXiv AI
Oct 31
Always Learning, Always Mixing: Efficient and Simple Data Mixing All the Time
The paper introduces OP‑Mix, an on‑policy data mixing algorithm that operates continuously across pretraining, mid‑training, and instruction tuning. By simulating candidate mixtures through low‑rank adapters trained on the current model, OP‑Mix avoids separate proxy models and remains grounded in the model’s learning dynamics. Experiments show OP‑Mix consistently finds near‑optimal mixtures, improving pretraining perplexity by 6.3% and reducing compute for continual learning by 66–95% compared to baselines.
arXiv Computation and Language
Oct 22
Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
The paper studies how long‑context continual pre‑training (LCCP) scales with token budget and model size. An empirical additive power law is derived that predicts LCCP performance across five model scales and over a hundred budget configurations, and a new Constant‑Last4 protocol is introduced for stable monitoring. Experiments show that performance continues to improve without clear saturation, even after 128K context extension and fine‑tuning, indicating that scaling LCCP yields transferable gains for downstream tasks.
arXiv Machine Learning
Jul 7
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
arXiv:2607. 04969v1 Announce Type: new Abstract: The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency.
Hugging Face Trending Papers
Oct 21
Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this qu...