← Back to all news
arXiv Machine Learning By Thiziri Nait Saada, Louis Bethune, Michal Klein, David Grangier, Marco Cuturi, Pierre Ablin

Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

Read the original on arXiv Machine Learning →

arXiv:2510. 00866v3 Announce Type: replace Abstract: Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 9

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.

By Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi
llmsragbenchmarks
More like this →
arXiv Machine Learning
Jul 31

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

arXiv:2604. 13977v2 Announce Type: replace-cross Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent.

By Joel Niklaus, Atsuki Yamaguchi, Michal \v{S}tef\'anik, Guilherme Penedo, Hynek Kydl\'i\v{c}ek, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf
llms
More like this →
arXiv Computation and Language
Oct 12

Host-Aware Web Crawling for LLM Pretraining Data

The study examines how to allocate web‑crawling resources to gather higher‑quality data for large language model pretraining. By combining document‑level quality scores, web‑graph centrality, and existing crawl coverage, it finds that most variation in quality is between hosts and that sampling a few pages from a host can predict the quality of the rest. The results suggest that depth‑based sampling of high‑scoring hosts yields far more high‑quality pages per fetch than random or centrality‑based strategies, while overall host coverage shows little correlation with quality.

By Malte Ostendorff
llms
More like this →