Google Joins OpenAI and Anthropic in Promising New Scraping Controls
Google Joins OpenAI and Anthropic in Promising New Scraping Controls
Google, OpenAI, Microsoft, and Anthropic have committed to honoring a new IETF robots.txt directive that allows publishers to opt out of generative AI training scraping with a single common signal. This shared policy token simplifies website management by replacing the need to configure separate rules for individual AI crawlers.
Major technology companies including Google, OpenAI, Microsoft, and Anthropic have agreed to support a new standardized robots.txt directive designed to help web publishers control how their content is used. Proposed by Anthropic technical lead Kevin Kelley through the Internet Engineering Task Force, the initiative introduces a shared policy token that lets website operators manage AI training preferences across multiple platforms at once. Currently, publishers must manually identify individual crawlers and configure separate rules for every provider. The upcoming common signal aims to streamline this process, though it only applies to future data collection and cannot undo past scraping activities.
Unified Controls for Web Publishers
Website owners currently face a cumbersome process when managing digital content rights, requiring them to identify individual crawlers and add separate rules for each distinct AI provider. The newly adopted proposal under consideration at the Internet Engineering Task Force seeks to eliminate this fragmented approach by establishing a single common signal.
By implementing this shared policy token within standard robots.txt files, publishers can efficiently communicate their AI training preferences across multiple technology platforms simultaneously without managing custom configurations for every single crawler.
Single Common Signal
Replaces fragmented rules by allowing publishers to apply AI training preferences across multiple platforms simultaneously.
Robots.txt Integration
Leverages familiar website configuration files that have long signaled which web crawlers can access a site.
How the New Directive Operates
Under the proposed framework, publishers utilize a specific user agent designated for AI training within their robots.txt configuration files. This setup allows website operators to entirely block data collection for model training while retaining the flexibility to permit access to specific folders, such as press kits.
Crucially, restricting data harvesting through this new user agent does not prevent a website from appearing in traditional search engine results. For instance, Google will interpret the instruction specifically as a stop sign for its Gemini training crawler without hurting general search visibility.
Granular Permissions
Website operators can block general AI training while selectively allowing access to specific directories like press kits.
Search Visibility Kept
Enabling the new scraping restrictions does not prevent web pages from appearing in standard search results.
Industry Adoption Timelines
Major artificial intelligence laboratories and platform providers have begun committing to support the upcoming standard with varying implementation schedules. Anthropic has already started supporting the new directive on its platforms, while OpenAI has pledged to implement the signal within the coming months.
Meanwhile, both Google and Microsoft have announced plans to introduce support for the standard in early 2027. Microsoft additionally noted that its implementation will provide granular controls allowing publishers to opt out of both AI training and AI grounding processes.
Anthropic Deployment
The company behind the proposed standard has already started supporting the new directive on its systems.
Microsoft and Google Schedule
Both technology giants have scheduled their official support for the common signal to launch in early 2027.
Limitations and Scope of the Standard
Despite the broad industry backing for the upcoming scraping controls, the standard contains inherent limitations regarding historical data retention. The directive cannot undo any past data collection that technology companies have already performed prior to implementation.
If an AI provider has previously crawled a specific web page and stored its contents, those materials remain within their training datasets. The new framework strictly serves to prevent unauthorized web scraping or data gathering across websites and specific pages in the future.
Why It Matters
The collaborative adoption of a unified robots.txt directive marks a pragmatic shift in how the technology industry addresses publisher content concerns. By moving away from fragmented, provider-specific opt-out rules toward a standardized Internet Engineering Task Force proposal, major AI developers are reducing friction for web operators. While the mechanism does not address retrospective data harvesting, establishing a transparent common signal creates a clearer operational boundary between search indexing and generative model training across the web ecosystem.
Google joins OpenAI and Anthropic in promising new scraping controls (engadget.com) · The biggest names in AI are backing a new way to opt out of scraping (theverge.com)
Reporting reviewed through October 31, 2026.
