arXiv AI
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 9
Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection
The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.
arXiv Computation and Language
Sep 22
English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
arXiv:2604.13286v2 Announce Type: replace Abstract: Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing...
arXiv AI
Aug 17
Scaling Domain Data Repetition in LLM Pretraining
arXiv:2608. 14071v1 Announce Type: new Abstract: As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)).
arXiv Machine Learning
Sep 11
Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
The paper proposes an information-weighted cross‑entropy loss that rescales token contributions using TF‑IDF statistics, thereby emphasizing semantically informative tokens and down‑weighting ubiquitous ones. Experiments on five decoder‑only language models (1.1B–13B parameters) show consistent reductions in memorized substring length while maintaining perplexity and downstream performance. The method is architecture‑agnostic, adds less than 3% computational overhead, and can be integrated into existing training pipelines.
arXiv AI
Sep 1
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
arXiv:2608.30609v1 Announce Type: cross Abstract: Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to addres...
arXiv AI
Jul 28
CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.