Host-Aware Web Crawling for LLM Pretraining Data

The study examines how to allocate web‑crawling resources to gather higher‑quality data for large language model pretraining. By combining document‑level quality scores, web‑graph centrality, and existing crawl coverage, it finds that most variation in quality is between hosts and that sampling a few pages from a host can predict the quality of the rest. The results suggest that depth‑based sampling of high‑scoring hosts yields far more high‑quality pages per fetch than random or centrality‑based strategies, while overall host coverage shows little correlation with quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms