arXiv AI
Jun 3
Pretraining Language Models on Historical Text
arXiv:2606. 02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.
A Language Model from 1913: Pretraining on Historical Text
Related stories
arXiv AI
Jul 15
Scaling Point-in-Time Language Models
arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.
arXiv Computation and Language
Sep 22
Chronologic: Measuring Language Models' Ability to Represent the Past
arXiv:2609.23178v1 Announce Type: new Abstract: Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its response...
arXiv AI
Jul 9
Co-LMLM: Continuous-Query Limited Memory Language Models
arXiv:2607. 07707v1 Announce Type: cross Abstract: Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights.
arXiv AI
Sep 2
KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
arXiv:2609.00082v1 Announce Type: cross Abstract: LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources su...
arXiv Computation and Language
Sep 11
CHRONOBERG: Capturing Language Evolution and Temporal Awareness in Foundation Models
CHRONOBERG is a temporally structured corpus of English book texts covering 250 years, curated from Project Gutenberg and enriched with temporal annotations. It enables quantification of lexical semantic change via time‑sensitive Valence‑Arousal‑Dominance analysis and the creation of historically calibrated affective lexicons. Experiments show that language models trained sequentially on CHRONOBERG struggle to encode diachronic shifts, highlighting the need for temporally aware training and evaluation pipelines.
arXiv AI
Jun 29
A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts
arXiv:2606. 27881v1 Announce Type: cross Abstract: Temporal variation poses a unique challenge for named entity recognition (NER) in historical texts, where entities drift in surface form and salience across time.
arXiv AI
Sep 10
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
arXiv:2609.07798v1 Announce Type: cross Abstract: Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, polic...
arXiv AI
Aug 26
Relative Time Intervals Representation for Word-level Timestamping with Masked Training
The paper introduces a method for improving fine-grained, temporally aligned outputs in Speech Large Language Models (SpeechLLMs) by replacing absolute timestamps with relative timestamps, which reduces vocabulary size and enhances generalization. It proposes a hybrid fine‑tuning strategy that fully fine‑tunes the timestamp‑augmented embedding layer and language model head while applying LoRA to decoder layers, and introduces a masked timestamp training objective to prevent over‑reliance on ground‑truth timestamps. Experiments show significant gains in timestamp prediction accuracy without compromising transcription quality.
arXiv Computation and Language
Sep 18
LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations
LMEnt is a toolkit designed to study how language models acquire and represent world knowledge. It includes a richly annotated pretraining corpus, an improved entity-based retrieval method, and twelve pretrained models with numerous checkpoints. The suite enables controlled experiments linking entity mentions in training data to downstream knowledge performance, revealing how co-occurrence patterns influence learning and editing difficulty.
Hugging Face Trending Papers
Jun 4
Can LLMs Be Constrained to the Past? Improving Knowledge Cutoff through Recall-Based Prompting
Prompted knowledge cutoff instructs a large language model (LLM) to act as if information beyond a specified cutoff date were unavailable. However, prior work mainly relies on direct-answer generation, which struggles when post-cutoff knowledge is not explicitly queried but is only causally related to the question.
arXiv AI
Jun 16
Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift
arXiv:2606. 15734v1 Announce Type: cross Abstract: Continual post-training enables models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can accumulate weight drift, potentially causing catastrophic forgetting and degrading general capabilities.