← Back to all news
arXiv AI By Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger

Identifying AI Web Scrapers Using Canary Tokens

Read the original on arXiv AI →

The paper introduces a method to detect which web scrapers feed data to large language models (LLMs) by deploying dynamic websites that issue unique canary tokens to each scraper. By querying LLMs for information about these sites, the authors can identify when an LLM consistently outputs the unique tokens, indicating exposure to a specific scraper. Experiments on 22 production LLM systems show the technique reliably uncovers both known and undisclosed scrapers, offering a tool for third parties to monitor and control unwanted web scraping.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 28

DeGenTWeb: A First Look at MGT-dominant Websites

DeGenTWeb is a system that identifies websites whose informational prose is largely machine‑generated text (MGT) with minimal human input. By adapting MGT detectors for web pages and aggregating results across sites, the study reports that 20.9% of Common Crawl‑archived sites (Jan–Jul 2026) and 13.6% of Bing‑linked sites (2025) are MGT‑dominant, with a 0.12% mislabeling rate for human‑authored sites. The prevalence of MGT‑dominant sites is increasing over time and many share the same owner.

By Sichang Steven He, Calvin Ardi, Ramesh Govindan, Harsha V. Madhyastha
llms
More like this →
arXiv Computation and Language
2d ago

When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition

The study investigates how often web agents incorrectly treat an HTTP 200 response as a successful fetch of useful content. By sampling 10,307 URLs—including random paths and expert‑verified claim citations—the authors found that roughly 20–25% of 200 responses lack the page’s main content, leading to inflated estimates of claim verifiability and citation drift. They propose simple heuristics and classifiers to detect unusable responses, achieving high recall with low false‑alarm rates, and recommend that agents report fetch outcomes explicitly.

By Shuyang Zhang (The Hong Kong Polytechnic University)
llmsagents
More like this →