arXiv AI
A Language Model from 1913: Pretraining on Historical Text
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 3
Pretraining Language Models on Historical Text
arXiv:2606. 02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.
arXiv AI
Jul 15
Scaling Point-in-Time Language Models
arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.
arXiv Computation and Language
Sep 22
Chronologic: Measuring Language Models' Ability to Represent the Past
arXiv:2609.23178v1 Announce Type: new Abstract: Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its response...
arXiv AI
Jul 9
Co-LMLM: Continuous-Query Limited Memory Language Models
arXiv:2607. 07707v1 Announce Type: cross Abstract: Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights.
arXiv AI
Sep 2
KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
arXiv:2609.00082v1 Announce Type: cross Abstract: LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources su...
arXiv Computation and Language
Sep 11
CHRONOBERG: Capturing Language Evolution and Temporal Awareness in Foundation Models
CHRONOBERG is a temporally structured corpus of English book texts covering 250 years, curated from Project Gutenberg and enriched with temporal annotations. It enables quantification of lexical semantic change via time‑sensitive Valence‑Arousal‑Dominance analysis and the creation of historically calibrated affective lexicons. Experiments show that language models trained sequentially on CHRONOBERG struggle to encode diachronic shifts, highlighting the need for temporally aware training and evaluation pipelines.