Saying no to AI training is about to take one line. Make sure it doesn't also say no to citations.
Google, OpenAI and Microsoft now back one shared robots.txt line for opting out of AI training. Training and citation are separate decisions. Here's what each company said, the Gemini catch, and how to decide on purpose.
By James Brady
Chief AI Officer, Product & AI Operations
Cloud AI cites who it can see. On October 30, three of the companies doing the seeing agreed on a simpler way to tell them what they can learn from.
Google, OpenAI and Microsoft each announced support for a shared AI training signal in robots.txt, based on work in the IETF's AI Preferences working group. One rule, read by every participating provider, instead of a separate block for each company's crawler.
That's a real improvement. It also creates a new way to make an expensive mistake. Training a model and citing your page in an answer are two different uses of your content. The new signal is about the first. If you're a business that wants to be recommended by AI, the second is the one you're paying attention to.
The short version
- There's a shared "no training" line coming. It's a robots.txt group with the user-agent
__usage_ai-training. Google says it aims to support it "in 2027." OpenAI says "in the coming months." Microsoft starts honoring its version on February 1, 2027. - Training and grounding are different jobs. Microsoft now lets you set them independently: training in robots.txt or per page, grounding (content used to inform an AI answer) per page only.
- Microsoft says blocking grounding costs visibility. In its words, choosing
DisalloworDisallowExtendedfor grounding means "you will see reduced brand visibility in AI experiences." - Google has a Gemini catch. Google will read the shared opt-out as a
Google-Extendeddisallow when you haven't setGoogle-Extendedyourself.Google-Extendedcovers Gemini training and grounding in Gemini Apps. - None of this touches classic search eligibility. Google says
Google-Extendeddoesn't affect Search inclusion or ranking. Microsoft says these preferences don't change eligibility for Bing results. - Nothing ships today. It's an announcement and an Internet-Draft. Decide now, write it down, and change your file when each provider says it's live.
What each company actually said
Google. In the Search Central blog post (October 30, posted by John Mueller), Google says that when a site "uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended." The same post adds: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Google's crawler documentation, updated the same day, says it "intends to support the AIPREF shared machine-readable training preference signal (with 'user-agent: __usage_ai-training') in early 2027." Source: Google Search Central, Supporting a shared AI training control for websites and Google's common crawlers list.
OpenAI. OpenAI says it "will honor a common signal in robots.txt" for "foundational generative AI training," aims to implement it "in the coming months," and that the new signal "will not replace the controls we already provide through robots.txt." Source: OpenAI, Making it easier for websites to express AI training preferences.
Microsoft. Microsoft went further. It adopts the shared training preference, extends it to the page level, and adds a separate page-level AI Grounding preference with three settings: Allow (the default), DisallowExtended (only Microsoft-operated experiences such as Copilot and Microsoft 365, not grounding services it sells to third parties), and Disallow. Its own example is the setup most businesses probably want: "you can set AI Training to Disallow and AI Grounding to Allow to keep content eligible to inform generated responses without allowing it to train generative models."
Source: Microsoft Bing, Your content, your choice: clearer preferences for training and grounding.
The standard itself. The syntax is described in an individual Internet-Draft, Machine-Readable Training Preference Signal, version 00, posted October 30. The IETF page says plainly that it "is not endorsed by the IETF and has no formal standing in the IETF standards process" yet.
The Gemini catch
Read Google's two pages together.
The announcement says a shared training opt-out, with no Google-Extended rule of your own, will be treated as a Google-Extended disallow. The crawler page says Google-Extended controls whether Google-crawled content can be used for training future Gemini models and "for grounding" in Gemini Apps and Grounding with Google Search on Vertex AI.
So, as written today, a site that adds only the shared "no training" line could also step out of Gemini grounding. Google's way to keep it is in the same post: explicitly allow Google-Extended, which the post says keeps you "eligible to appear in responses and receive clicks via grounding in Gemini Apps." That also means allowing Gemini training through that token. Google doesn't currently offer a robots.txt split between Gemini training and Gemini grounding.
That's my reading of Google's own words, not a statement Google made in one sentence. Recheck it when Google publishes implementation details.
Why this matters more than it looks
A lot of businesses will hear "there's finally one line to stop AI training on your site" and add it the same afternoon. Reasonable instinct. But it's worth knowing what you're trading.
- Microsoft keeps the two apart. Training no, grounding yes is a supported setup.
- Google ties Gemini grounding to the same token the shared opt-out maps to. AI Overviews and AI Mode are governed separately, through the Search generative AI control in Search Console (covered in robots.txt is a door, not a usage policy).
- OpenAI says its existing crawler controls keep working alongside the new signal. ChatGPT search visibility still runs through
OAI-SearchBot, which is separate from training.
The pattern is the same everywhere: being learned from and being cited are separate decisions. Make them separately.
Decide on purpose (about 15 minutes)
- Write down your goal in one sentence per use. "Train on us: yes/no. Use us to answer questions and link back: yes/no." For most local and service businesses I talk to, the second answer is yes.
- Open your live robots.txt. Note any rule for
GPTBot,OAI-SearchBot,Google-Extended,Bingbot, or a blanket "block all AI bots" block. The Oct 23 post maps each token to the product it changes. - If you use
NoArchiveorNoCacheon Bing, put a date on the calendar. Microsoft says new preferences take effect as it crawls from February 1, 2027; after March 1, 2027, those old tags "will no longer be read as a preference for training or grounding." Sites using them as of December 15, 2026 can record a temporary grounding preference in Bing Webmaster Tools. - If your answer is "no training, yes citations," plan the pair, not the single line: the shared
__usage_ai-trainingdisallow, plus (for Google, if you want Gemini grounding) an explicitGoogle-Extendedallow, plus leaving Microsoft grounding at its defaultAllow. Accept that the Google allow also permits Gemini training. - Don't edit anything for Google or OpenAI yet. Neither says it's honoring the signal today. Add it when each confirms, then check your answers again on a fixed set of prompts.
What this is not (honesty block)
- Not live yet. Google targets 2027 ("early 2027" on its crawler page). OpenAI says "coming months" with no date. Microsoft's date is February 1, 2027.
- Not a final standard. It's an individual Internet-Draft, version 00, without formal IETF standing.
- Not a grounding control from Google or OpenAI. Neither announced a separate grounding preference. Only Microsoft did.
- Not a promise either way. Microsoft: "Neither choice guarantees inclusion, traffic, or payment." Allowing grounding makes you eligible; it doesn't make you cited.
- Not a crawl rule. Microsoft says these settings "govern use of content, not crawl access," and a setting only counts once its crawler sees it on the page.
- Not legal advice. Whether to allow training is a business and licensing question. If it touches contracts or licensing, talk to counsel.
- Other providers. Trade press reports more companies joined. I only cite the three whose own pages I read.
Related New Reward Field Guides
- robots.txt is a door, not a usage policy. Know which AI bot you're actually blocking.
- Google is signing some agent requests. That is identity, not a citation.
- Cloudflare is helping ChatGPT find more pages. Found is not cited.
- Most AI citations come from pages you don't own. Your own pages still have a job.
Want the fast read first?
I'm not going to pretend a scan is a strategy. It's a free check of how visible your business is across the AI engines buyers use, a 0–100 score in about two minutes, email only, no card. Get your AI-visibility score.
FAQ
Common questions
Will the shared training opt-out remove me from Google Search or AI Overviews?
Google says `Google-Extended` "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode have their own control in Search Console. Gemini Apps grounding is the part to watch.
Is blocking AI training bad for visibility?
Not by itself, per Microsoft, which lets you block training and keep grounding. With Google, the shared opt-out also maps to the token that controls Gemini Apps grounding unless you set that token yourself.
Do I need to do anything today?
Only decide and write it down. If you use `NoArchive` or `NoCache` for Bing, the March 1, 2027 deadline is the real one.
Does allowing grounding guarantee I get cited?
No. Microsoft says neither choice "guarantees inclusion, traffic, or payment." It keeps you eligible. Being worth citing is still the work.
Perplexity
Grok