arXiv:2510. 00866v3 Announce Type: replace Abstract: Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential.
arXiv:2604.28075v3 Announce Type: replace-cross Abstract: Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. Ho...
The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.
arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.
arXiv:2603.12165v3 Announce Type: replace Abstract: Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult...
The study examines how to allocate web‑crawling resources to gather higher‑quality data for large language model pretraining. By combining document‑level quality scores, web‑graph centrality, and existing crawl coverage, it finds that most variation in quality is between hosts and that sampling a few pages from a host can predict the quality of the rest. The results suggest that depth‑based sampling of high‑scoring hosts yields far more high‑quality pages per fetch than random or centrality‑based strategies, while overall host coverage shows little correlation with quality.
arXiv:2606. 11499v1 Announce Type: cross Abstract: The performance of modern language models depends critically on pretraining data composition.
LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.
arXiv:2610.33969v1 Announce Type: new Abstract: "Data quality" is widely invoked in instruction tuning, but the term is often underspecified. We study it as an operational property by asking which si...
RePro is a web‑recycling technique that trains a small language model (as little as 1 B parameters) with reinforcement learning to produce high‑quality, faithful rephrasings of pretraining data. The method uses one quality reward and three faithfulness rewards to preserve core semantics and structure while converting organic data into better training examples. Experiments show that RePro boosts downstream accuracy by 3.7–14.5 % over organic‑only baselines and improves data efficiency 2–3×, outperforming prior prompting‑based recycling approaches.
MAP4CS is a data pruning framework that selects a small, high‑quality core subset from large code corpora by combining syntactic structure, semantic diversity, and distributional representation, followed by rule‑based filtering. Experiments on two large datasets show that using only 5% of the training data, MAP4CS outperforms random sampling and matches or exceeds full‑corpus fine‑tuning, supporting the idea that less data can be more effective. The method also automatically removes duplicates and noise, producing a lexically diverse and information‑dense training set.
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.