OpenAI, Google, Microsoft, and Anthropic Will Honor One robots.txt No-AI-Training Token. Only Microsoft Has Given a Date: February 1, 2027.
On October 30 the four labs said they will honor a shared robots.txt preference from the IETF AIPREF draft: a User-Agent of __usage_ai-training with a Disallow line. Microsoft will honor it as it crawls from February 1, 2027, and will stop reading NoArchive and NoCache for training and grounding after March 1, 2027. Google aims to support it in 2027 and will treat the shared opt-out like a Google-Extended disallow where none is set. OpenAI says the coming months. Anthropic's commitment is press-only so far. The token is still a draft, nobody honors it today, and it does not undo past scraping. We read the OpenAI, Google, and Bing posts.
By T.W. Ghost
Four labs said today, October 30, 2026, that they will honor a single robots.txt preference for AI training, drawn from the IETF's AIPREF draft. The mechanism is a User-Agent line of __usage_ai-training with a Disallow, in Microsoft's published examples. A publisher who adds it is saying, once, that the site's content is not for training models, instead of maintaining a separate block for GPTBot, Google-Extended, and every other crawler name.
Nobody honors it today. Microsoft is the only lab that printed a date: it will honor the new preference as it crawls from February 1, 2027, and after March 1, 2027 it will stop treating NoArchive and NoCache as signals for training or grounding. Google aims to support the token "in 2027," and will treat the shared disallow as a Google-Extended disallow when a site has not set Google-Extended explicitly. OpenAI says it will implement "in the coming months," with existing crawler controls staying in force. Anthropic is reported to have started implementing, but we found no dated Anthropic page saying so. The token is a draft, it is forward-looking, and nothing about it reaches back into the data the labs already have. We read OpenAI's post, Google's Search Central post, and the Bing post. Here is what held up.
Who Committed to What
| Lab | Commitment | Date | Source |
|---|---|---|---|
| Microsoft | Honor __usage_ai-training as crawled; NoArchive and NoCache no longer read for training or grounding | February 1, 2027; March 1, 2027 | Bing blog |
| Support the shared preference; treat it as Google-Extended disallow where Google-Extended is unset | "In 2027" | Google Search Central | |
| OpenAI | Implement; existing crawler controls remain | "In the coming months" | OpenAI post |
| Anthropic | Reported to have started implementing | None | Press only |
| Meta | Voiced support | None | Press only; no commitment found |
| All | Traditional search ranking unaffected (Bing, Google) | Vendor statements |
One date, from one lab. February 1, 2027 is a Microsoft date. Every other commitment is a season or a phrase. A publisher reading "the industry now honors a no-training flag" should read the table instead.
Grounding and training are different switches at Bing. Microsoft keeps separate page-level preferences for AI grounding, the retrieval that feeds answers, apart from training. The shared token covers training. Check both.
Past scrapes stay scraped. The preference governs future crawls by labs that have implemented it. It is not a deletion request, and no lab has described it as one.
At Google, the opt-out has a cost. A Google-Extended disallow also affects a site's eligibility for Gemini grounding, by Google's own description, so the shared token inherits that trade-off where Google-Extended is not set separately.
Verified vs Unconfirmed: The Scorecard
| Claim | Verdict |
|---|---|
| Shared robots.txt AI-training preference announced Oct 30 by OpenAI, Google, Microsoft | Verified (three vendor posts) |
Token __usage_ai-training with Disallow | Verified (Microsoft's published examples) |
| Microsoft honors from Feb 1, 2027; NoArchive and NoCache not read for training or grounding after Mar 1, 2027 | Verified (Bing blog) |
| Google "in 2027"; shared opt-out treated as Google-Extended disallow where unset | Verified (Google Search Central) |
| OpenAI "in the coming months"; existing controls remain | Verified (OpenAI post) |
| Anthropic implementing | Press (no Anthropic-owned page found this pass) |
| Meta commitment | Unverified (press support, no commitment) |
| Search ranking unaffected | Vendor statements (Bing, Google) |
| IETF AIPREF status | Draft |
What To Do
Publish the token now if you do not want your content used for training; Microsoft will read it on crawl after February 1, and the others when they ship.
Keep every vendor-specific block you already have until the vendor behind it names a date; the shared token replaces nothing yet.
The frontier is elsewhere: this is the first thing the four labs have agreed to honor in common, and it starts next year. Our model comparison has the current standings on what they built in the meantime, and the policy that is already dated this quarter is OpenAI's November 20 usage-policy change.
Sources
- OpenAI: helping websites express AI training preferences
- Google Search Central: AI training control
- Bing: your content, your choice, clearer preferences for training and grounding
*Not sure which lab's crawler is reading your site and for what? Take the free 2-minute quiz and get matched. Then read the other default that flipped this week, on the API side: Gemini API logging for new paid projects.*