← Back to all news
arXiv Computation and Language By Yuanjian Xu, Jianing Hao, Tianze Sun, Yongyao Duan, Sam Chen, Jinyi Han, Wanbo Zhang, Yuxuan Sun, Guang Zhang, Zhong Li

Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

Read the original on arXiv Computation and Language →

The paper reviews data strategies for large language models, emphasizing that data strategy—selecting, weighting, and scheduling training data—determines model capabilities within a fixed data budget. It treats high‑quality data as having training‑dependent utility, varying with the target objective, model, and training state. The authors organize existing methods by decision granularity, model dependence, and temporal adaptivity, comparing their computational costs and empirical support, and note limited evidence for transfer across model scales and end‑to‑end cost parity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
5d ago

Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings

The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.

By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
llmsragfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Jul 7

Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

arXiv:2607. 04969v1 Announce Type: new Abstract: The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency.

By Jingwei Zuo, Cong Zeng, Ilyas Chahed, Maksim Velikanov, Dhia Eddine Rhaiem, Pasquale Balsebre, Abhay Kumar, Younes Belkada, Hakim Hacid
llms
More like this →
arXiv AI
Jul 15

Scaling Point-in-Time Language Models

arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.

By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
llmsfine-tuningbenchmarkssafety
More like this →