Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings

The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• rag
• fine-tuning
• benchmarks
• safety