🔍 Read the full analysis: OpenAI, Google, Microsoft And Anthropic Align On An AI Training Opt-Out on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal that lets website operators state they do not want their content used to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live web content, and will be adopted on different timelines; Meta expressed support but did not commit to honoring it.
OpenAI, Google, Microsoft and Anthropic said they will support a shared robots.txt signal allowing website operators to ask that their content not be used to train generative AI models. The announcements, made October 30, point to the same Internet-Draft, but the signal is a voluntary preference rather than a technical block, and the companies have set different implementation timelines.
The proposed rule uses the user-agent label __usage_ai-training followed by Disallow: /. A site owner could add the group to its robots.txt file to express a preference against training use. The text is in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley on October 30. It is not an adopted internet standard.
Anthropic says it has started implementation, though it says support will take time to reach its services. OpenAI said it aims to implement the signal “in the coming months” for foundational generative AI training. Google aims for 2027; its crawler documentation describes the target as early 2027. Google says that, if a site uses the shared signal but has no Google-Extended rule, it plans to treat the shared preference as a disallow for Google-Extended.
Microsoft plans to honor the training preference from February 1, 2027, as it crawls websites. It is also introducing a separate page-level preference for “AI Grounding,” which covers content used to inform AI answers in Bing, Copilot and Microsoft 365. Microsoft says the new preferences replace NoArchive and NoCache for training or grounding purposes after March 1, 2027. Existing crawler-specific rules, including GPTBot, Google-Extended and ClaudeBot, remain relevant during the rollout.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
The agreement gives publishers and other site operators a single requested signal to communicate a training preference to several major AI providers, rather than relying only on separate company-specific crawler rules. That may simplify policy management as firms begin to recognize the same instruction. Its practical value, however, depends on each company implementing and honoring it.
The distinction between a preference and enforcement matters. The rule does not stop a crawler from accessing a site, remove material from models already trained, or create a payment arrangement. It also does not, by itself, prevent a company from using current web content to answer a question. Microsoft’s separate AI Grounding control addresses that use within its products, but the shared training signal is specifically about model training.
For publishers, the announcements create a reason to review their robots.txt policies and decide separately how to handle training, AI-generated answers and search. Microsoft says its controls do not change eligibility for Bing search results, while Google says Google-Extended does not affect Google Search. The new signal therefore should not be read as a general instruction to remove a site from search or from every AI product.
Products Worth Considering
Bose QuietComfort Wireless Noise Cancelling Headphones, Sandstone
Samsung Galaxy Tab S10 FE 128GB Wi-Fi Android Tablet, Blue
Sonos Era 100 SL Compact Microphone-Free Speaker – Black
As an affiliate, we earn on qualifying purchases.
How the Draft Fits Existing Rules
Before this proposal, website operators commonly used separate directives for individual companies. OpenAI’s GPTBot, Google-Extended and Anthropic’s ClaudeBot each have their own crawler controls. Microsoft’s NoArchive and NoCache tags have also been used for preferences affecting its systems. A publisher seeking to express the same position to multiple firms has had to manage more than one rule and track changes to each provider’s policy.
The shared token comes from work associated with the IETF’s AI Preferences working group, which considers machine-readable ways to express preferences about online content. The draft is one contribution to that process, not a final IETF specification. At a working-group meeting on October 21, one of the group’s chairs said, “We haven’t declared consensus on anything yet.” The draft’s publication and company announcements therefore establish support for an approach, not formal standards approval.
Company commitments also differ in scope and timing. Anthropic’s statement describes implementation work already begun; OpenAI gives a broad target of coming months; Google gives a 2027 goal; and Microsoft names a February 2027 start date. Meta spokesperson Andy Stone said the company supports publishers’ ability to express preferences and called a common signal an important step, but did not say Meta would honor it.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Remain Open
Whether every company will follow the preference is not guaranteed. The signal is voluntary, and the announcements do not establish an enforcement process or a remedy if a company disregards it. Meta has voiced support but has not made a specific implementation commitment. The source material also reports no statement from Apple, Amazon, xAI or Perplexity; that does not establish what those companies may decide later.
The final status of the proposal is also unsettled: it remains an individual Internet-Draft with no formal standing in the IETF standards process. The companies’ intended dates may change, and their services may not all support the signal at once. The rule’s scope is limited to training preferences; it does not cover AI answers from live web content unless a provider offers a separate control, as Microsoft has announced for its products.
It is not yet clear how participating companies will handle every site configuration or how reliably operators can verify that a preference was applied across services. The stated plans describe intended treatment of the signal, not an independent audit of compliance.
Watch the Rollout Dates
Website operators can add the shared training group if they want to state that preference, but should keep relevant company-specific rules during the transition until each provider’s support is active. Operators using Microsoft’s NoArchive or NoCache tags for training or grounding preferences should review the announced change: Microsoft says those tags stop counting for those purposes after March 1, 2027, with its new controls taking effect from February 1 as it crawls sites.
The next milestones are Anthropic’s expansion of support across its services, OpenAI’s planned implementation in the coming months, and the 2027 rollouts from Google and Microsoft. Further company statements, changes to crawler documentation and progress within the IETF working group will clarify how broadly the signal is adopted. Until then, the shared line is a request that participating companies say they will honor, not a universal or technically enforced opt-out.
Key Questions
What does the shared AI training opt-out do?
It lets a site operator add a shared robots.txt preference asking participating companies not to use the site’s content to train generative AI models. The proposed group uses User-Agent: __usage_ai-training and Disallow: /.
Does the signal block AI companies from crawling a website?
No. The signal expresses a preference about training use; it is not a technical barrier to crawling. It also does not remove content from models that have already been trained.
Does it stop AI systems from using a site in answers?
Not by itself. The shared proposal concerns training, not AI answers generated using live web content. Microsoft has announced a separate AI Grounding preference for content used in Bing, Copilot and Microsoft 365.
When will the companies support the signal?
Anthropic says implementation has started, OpenAI aims for the coming months, Google targets 2027 and Microsoft says its preference takes effect from February 1, 2027 as it crawls. The companies’ timelines and service coverage differ.
Has Meta agreed to honor the opt-out?
Meta spokesperson Andy Stone expressed support for publishers stating preferences and for a common signal, but did not commit to honoring it in the statement described in the source material.
Source: ThorstenMeyerAI.com
