🔍 Read the full analysis: Your Guide To Setting The AI Training Opt-Out Across Services on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced support for a shared robots.txt signal asking participating services not to use a website’s content for generative AI training. The signal is voluntary, does not block crawling or AI answers, and the companies plan to adopt it on different schedules; Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared website preference asking them not to use content for generative AI training. The proposed robots.txt rule gives publishers one signal to use across participating services, but it is a voluntary request, not a technical barrier, and each company has set a different implementation timeline.
The signal appears in an Internet-Draft titled Machine-Readable Training Preference Signal, written by Anthropic standards lead Kevin Kelley and published October 30. A website can add a group for User-Agent: __usage_ai-training followed by Disallow: / to express a preference against training use. The draft builds on work in the Internet Engineering Task Force’s AI Preferences working group, but it is not an adopted internet standard.
The companies have announced different plans. Anthropic says it has started implementing support, while noting that rollout across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Google is aiming for 2027; its crawler documentation says early 2027. Microsoft says it will honor the shared training preference from February 1, 2027, as it crawls websites.
Microsoft also announced a separate page-level preference for AI Grounding, covering web content used to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect February 1, 2027. Its existing NoArchive and NoCache settings will stop counting as preferences for training or grounding after March 1, 2027. Existing controls such as GPTBot, Google-Extended and ClaudeBot remain available during the change.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, the shared rule could reduce the work of maintaining separate crawler instructions for each company. A site that wants to object to training can express that preference once for participating services, rather than relying only on individual company-specific rules. That is a practical change in how publishers communicate their choices, especially while the companies’ systems adopt the signal at different times.
But the rule does not stop a crawler from accessing a page. It communicates a preference that companies say they will honor; it is not an access-control mechanism or a guarantee that all AI providers will comply. Nor does it set terms for payment, promise that a site will appear in search or AI products, or remove content from models that have already been trained. Microsoft says its choices “Neither choice guarantees inclusion, traffic, or payment.”
Publishers should also separate three decisions: whether content may be used for model training, whether it may inform live AI answers, and whether it may be indexed for search. The new shared signal addresses training. Microsoft’s additional grounding preference addresses a separate use, while existing search settings remain distinct. Treating these controls as interchangeable could leave a publisher’s actual preference unclear.
Products Worth Considering
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
Jackery Explorer 300 Portable Power Station, 292Wh
As an affiliate, we earn on qualifying purchases.
How Company Controls Differ
Before this proposal, site operators generally had to use separate mechanisms to state preferences to different services. Examples include GPTBot rules for OpenAI, Google-Extended for Google, and ClaudeBot for Anthropic. Microsoft has also used page-level NoArchive and NoCache tags. The shared signal is intended to provide a common way to express a training preference, while the existing company-specific controls continue to matter during the rollout.
Google says that if a publisher uses the new token but has no Google-Extended rule, it plans to interpret the new preference as though the publisher had disallowed Google-Extended. Google says the new control will not affect Google Search. Microsoft says its training and grounding preferences do not change eligibility for Bing search results. OpenAI says its current crawler controls will continue to apply.
The proposal’s status is limited: it is an individual Internet-Draft, not a formal IETF standard. At an October 21 meeting of the AI Preferences working group, a chair said the group had not declared consensus on anything. The announcements therefore reflect company plans and commitments as stated by each company, not a final standards decision binding on all internet services.
““has already started implementing support for this draft””
— Anthropic standards lead Kevin Kelley
Who Will Honor the Rule
Participation beyond the four announcing companies remains unclear. Meta has expressed support for publishers’ ability to state preferences, but has not committed to honoring this signal. The source material reports no statement from Apple, Amazon, xAI or Perplexity; that does not establish whether they will adopt it later.
Implementation is also incomplete and staggered. Anthropic says rollout has begun but will take time; OpenAI has not given a specific date beyond “the coming months”; Google points to 2027, with documentation specifying early 2027; and Microsoft’s date is February 1, 2027. The precise handling of the signal by services that have not announced support, and how compliance will be monitored, are not established in the draft or announcements.
The rule’s scope should not be read more broadly than stated. It concerns training preferences, not all crawling, live retrieval or AI-generated answers. Microsoft has announced a separate grounding setting, but the shared training signal itself does not cover that use. It also cannot undo training that has already taken place.
Staggered Rollout Through 2027
Website operators can add the shared training preference if they want participating companies to receive it, while retaining company-specific rules until each service has implemented the common signal. Sites using Microsoft’s NoArchive or NoCache settings should review Microsoft’s announced timetable: those tags stop counting as training or grounding preferences after March 1, 2027, following the new settings’ February 1 start.
Publishers should check each provider’s crawler documentation for implementation updates and decide separately how they want to handle training, AI grounding and search. The IETF working group may continue discussing the draft, but the material provided does not establish when or whether it will reach formal standards status. Adoption by other AI companies is also still to be announced.
Key Questions
What does the shared AI training opt-out do?
It adds a robots.txt preference asking participating services not to use a website’s content for generative AI training. It is a request that companies say they will honor, not a mechanism that blocks access.
Which companies have announced support?
OpenAI, Google, Microsoft and Anthropic have announced support, with different implementation schedules. Meta has voiced support for publishers expressing preferences but has not said whether it will honor this particular signal.
Does the rule stop AI answers from using a website?
No. The shared signal covers training, not live use of web content to inform AI answers. Microsoft announced a separate AI Grounding preference for that use.
Does adding the rule erase content from existing AI models?
No. The preference does not remove content from models that have already been trained, and it does not guarantee payment, search visibility or traffic.
Should publishers remove their existing crawler rules?
Not based on the announcements provided. Existing rules such as GPTBot, Google-Extended and ClaudeBot remain relevant while companies roll out the shared signal. Microsoft separately says NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027.
Source: ThorstenMeyerAI.com
