arXiv AI
Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 31
How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.
arXiv Machine Learning
Aug 27
Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
arXiv:2608. 25826v1 Announce Type: cross Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched.
arXiv Machine Learning
Sep 23
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
arXiv:2609.26347v1 Announce Type: cross Abstract: The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP)...
arXiv AI
Oct 5
A Language Model from 1913: Pretraining on Historical Text
arXiv:2606.02991v2 Announce Type: replace-cross Abstract: While modern language models increasingly rely on ever-larger web corpora, we show that pretraining on historical text (e.g., pre-1913 text)...
arXiv AI
Sep 2
KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
arXiv:2609.00082v1 Announce Type: cross Abstract: LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources su...
arXiv Computation and Language
Oct 15
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
AFCS (Atomic‑Fact‑Conditioned Synthesis) is a new pipeline for generating synthetic data for language‑model pretraining that focuses on extracting and using fine‑grained atomic facts from source documents. Unlike previous methods such as WRAP and Nemotron‑CC, AFCS explicitly requires secondary entities, list/table entries, and precise temporal attributes to appear in the synthesized corpus, employing structure‑aware extraction, source‑grounded verification, local context expansion, and multi‑format generation. In experiments with OLMo‑3‑7B and a 32B model on the FineWiki corpus, AFCS outperforms the baselines on several QA benchmarks and improves general benchmark performance by +1.09 when integrated mid‑training with 100B tokens.