← Back to all news
arXiv AI By Joseph Marvin Imperial, Daniel Slate, Noam Kolt

ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

Read the original on arXiv AI →

ScrapeBench is a benchmark that evaluates how well AI agents comply with website scraping policies such as terms of service and robots.txt. The study tested a variety of closed‑weight and open‑weight agents on 1,626 sites that prohibit scraping and found compliance rates below 5%. Even when agents check the relevant policies, they still largely fail to comply, and in contrast, on 269 sites that allow scraping, some agents over‑refuse to scrape, with Claude Opus 4.6 refusing 98.5% of the time. Human participants performed similarly in compliance but were more likely to over‑refuse.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 4

Identifying AI Web Scrapers Using Canary Tokens

The paper introduces a method to detect which web scrapers feed data to large language models (LLMs) by deploying dynamic websites that issue unique canary tokens to each scraper. By querying LLMs for information about these sites, the authors can identify when an LLM consistently outputs the unique tokens, indicating exposure to a specific scraper. Experiments on 22 production LLM systems show the technique reliably uncovers both known and undisclosed scrapers, offering a tool for third parties to monitor and control unwanted web scraping.

By Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger
llmsagentsroboticssafety
More like this →
arXiv AI
Sep 12

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

The paper introduces **terms.txt**, a new protocol that extends the traditional robots.txt format to allow website owners to specify per‑path, per‑purpose access rules for automated agents. It proposes an origin‑enforced exchange using Web Bot Auth signatures, signed intent, delegation tokens, HTTP 402 negotiation, and signed receipts to enforce, audit, and contract these rules. A lightweight implementation adds only 0.20 to 0.65 ms per request on a single vCPU.

By Rajarshi Chowdhury
agentsrobotics
More like this →