On October 30, 2026, OpenAI, Google, Microsoft and Anthropic announced support for a shared robots.txt policy token, __usage_ai-training, that lets a website express a preference against generative AI training in a single rule. Support rolls out over the coming months and into 2027. Marketing and content teams that want to limit training use of their sites should add the token now and keep existing company-specific controls in place.
Four major AI providers agreed on October 30, 2026, to honor one shared robots.txt signal for AI training preferences. Instead of maintaining separate blocks for each company’s crawler, site owners can use the policy token __usage_ai-training. The proposal comes from an IETF Internet-Draft authored by an Anthropic engineer and published the same day. Google’s Search Central blog, OpenAI, and Microsoft each confirmed they will support the signal.
What changed?
A single robots.txt group using User-Agent: __usage_ai-training can now express a site-wide or path-specific preference against AI training that multiple providers have said they will honor.
When?
Announced October 30, 2026. Anthropic is already implementing; OpenAI targets the coming months; Microsoft starts February 1, 2027; Google aims for 2027.
Who is affected?
Publishers, brands, and any site owner that wants to control whether its content is used to train generative AI models.
What to do this week?
Decide whether to opt out of training. Add the shared token to robots.txt if yes, and retain existing GPTBot, Google-Extended, and other controls until full rollout.
What exactly was announced on October 30?
Google’s Search Central blog stated that the company plans to support a shared AI training control in robots.txt based on work from the IETF AIPREF working group. The same day, OpenAI, Microsoft, and Anthropic indicated they will honor the same token. The mechanism reuses ordinary robots.txt syntax: a group whose User-Agent line names the policy token rather than a specific bot. OpenAI’s blog post explained: “Today, we’re announcing that OpenAI will honor a common signal in robots.txt… We aim to implement the signal in the coming months… This new signal will not replace the controls we already provide through robots.txt.”
The underlying document is an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Kevin Kelley of Anthropic and dated October 30, 2026. It is not yet a formal IETF standard.
How does the robots.txt token work?
A typical opt-out looks like this:
User-Agent: __usage_ai-training Disallow: /
An Allow line can carve out exceptions (for example, a press kit). The signal covers training use only; it does not block crawling, search indexing, or AI grounding/answers. Existing company-specific rules (Google-Extended, GPTBot, ClaudeBot, etc.) continue to function alongside the shared token. The draft example also shows path-level exceptions:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/
When will each provider actually honor it?
Timelines differ. Anthropic has begun implementation. OpenAI said it will support the signal in the coming months while keeping its current crawler controls. Microsoft set a start date of February 1, 2027. Google aims to support the token in 2027 and has said that, where a site opts out via the shared token and does not explicitly allow Google-Extended, it will treat the site as having disallowed Google-Extended. The underlying IETF draft is an individual submission and is not yet a formal standard. Meta has not joined the shared-token effort.
Because rollout is staggered, a site that adds the token today will see the preference respected first by Anthropic, later by OpenAI, and still later by Microsoft and Google. Existing per-bot rules therefore remain the only reliable immediate control for the slower-moving providers.
What should marketing and content teams do this week?
Inventory current robots.txt rules. Decide whether the brand wants to opt out of training, allow training on selected paths, or leave the default (allow). If opting out, add the shared token immediately. Do not remove existing company-specific blocks until each provider confirms it is honoring the shared signal. Document the decision for legal and SEO stakeholders. Monitor the IETF draft and provider announcements through 2027. Test the robots.txt file after the change and keep a copy of the prior version for reference.
What the token does not do
It does not stop bots from crawling the site. It does not remove content already in existing training sets. It does not create a licensing or payment obligation. Providers remain free to ignore the signal; the announcements are voluntary commitments. It also does not control AI Overviews, ChatGPT citations, or other grounding/answer uses; those continue to be governed by separate crawler rules or other mechanisms.
Does the shared token replace Google-Extended or GPTBot?
No. Existing controls continue to work. The shared token is an additional, simpler way to express a preference.
If I add the token, will Google stop using my content for training immediately?
Google has said it aims to support the token in 2027. Until then, continue using Google-Extended if you want an immediate opt-out.
Does this stop AI Overviews or ChatGPT from citing my site?
No. The signal is limited to training. Crawling and grounding are controlled by other rules.
Is the IETF draft a finished standard?
No. It is an individual Internet-Draft dated October 30, 2026. Formal standardization is still in progress.
Has Meta joined the shared token?
No. Meta’s commercial-terms process is separate and the company has not announced support for __usage_ai-training.
Last Updated: November 2, 2026.
Related: Marketing hub · Existing posts on Meta commercial terms and robots.txt controls.
Discover more from Kurums | Business Intelligence
Subscribe to get the latest posts sent to your email.