← Back to all news
arXiv AI

DeGenTWeb: A First Look at MGT-dominant Websites

DeGenTWeb is a system that identifies websites whose informational prose is largely machine‑generated text (MGT) with minimal human input. By adapting MGT detectors for web pages and aggregating results across sites, the study reports that 20.9% of Common Crawl‑archived sites (Jan–Jul 2026) and 13.6% of Bing‑linked sites (2025) are MGT‑dominant, with a 0.12% mislabeling rate for human‑authored sites. The prevalence of MGT‑dominant sites is increasing over time and many share the same owner.

Related stories

arXiv AI
6d ago

The uneven rise of AI-generated web content

The study examines how generative AI has reshaped the public web in under five years, using over 5 million crawlable pages. It finds that AI-generated text has risen sharply but unevenly, with higher prevalence on English-language and .com sites, lower on .gov and .edu, and varying by a page’s centrality in the web graph. The authors employ multiple detection tools and a Bayesian hierarchical model to estimate prevalence while accounting for detector errors.

By Adam Visokay, Soumyakanti Pan, Molly Griston, Polina Iasakova, Mark Nepf, Denis Peskoff, Riya Sinha, Jikai Sun, Isabelle Thiara, Letao Wang, Yiwei Wu, Lauren Yan, Tyler H. McCormick
More like this →
arXiv AI
Sep 4

Identifying AI Web Scrapers Using Canary Tokens

The paper introduces a method to detect which web scrapers feed data to large language models (LLMs) by deploying dynamic websites that issue unique canary tokens to each scraper. By querying LLMs for information about these sites, the authors can identify when an LLM consistently outputs the unique tokens, indicating exposure to a specific scraper. Experiments on 22 production LLM systems show the technique reliably uncovers both known and undisclosed scrapers, offering a tool for third parties to monitor and control unwanted web scraping.

By Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger
llmsagentsroboticssafety
More like this →
arXiv Computation and Language
Oct 12

Host-Aware Web Crawling for LLM Pretraining Data

The study examines how to allocate web‑crawling resources to gather higher‑quality data for large language model pretraining. By combining document‑level quality scores, web‑graph centrality, and existing crawl coverage, it finds that most variation in quality is between hosts and that sampling a few pages from a host can predict the quality of the rest. The results suggest that depth‑based sampling of high‑scoring hosts yields far more high‑quality pages per fetch than random or centrality‑based strategies, while overall host coverage shows little correlation with quality.

By Malte Ostendorff
llms
More like this →
arXiv Computation and Language
Oct 27

RefinedGAW: A Refined German Academic Web Dataset

arXiv:2610.34572v1 Announce Type: new Abstract: The largest open German text datasets are built from broad web crawls, which provide scale but limited coverage of specific domains. We introduce Refin...

By Maximilian Idahl, Oleh Astappiev, Benedikt Droste, Robert Scholz, Bj\"orn Pl\"uster, Jan Philipp Harries, Robert J\"aschke, Wolfgang Nejdl
safety
More like this →
arXiv Computation and Language
Sep 22

The Corroboration Illusion: When More News Makes LLM Forecasts Less True

The paper demonstrates that large language models (LLMs) used for forecasting real‑world events can be manipulated by simply publishing new articles, even without direct access to the model or its retriever. By injecting a small number of targeted news pieces into a common crawl corpus, an adversary can flip over half of the forecast probabilities and significantly degrade forecast accuracy. The study also shows that common defense strategies can be cheaply bypassed, highlighting the vulnerability of probabilistic LLM judgments to information‑supply‑chain attacks.

By Yuan Lu, Yukuan Zhang
llmsrag
More like this →