robots.txt is a door, not a usage policy. Know which AI bot you're actually blocking.

Cloud AI cites who it can see. A lot of businesses decided what AI can see with one copy-pasted block in a file they haven't opened since.

This week Google said out loud what that file can and can't do. In a Search Central video published October 21, Gary Illyes put it plainly: "As defined by RFC 9309, robots.txt is an access-preference protocol. It only tells bots, 'You can crawl this page,' or, 'You shouldn't crawl this other page.' It doesn't dictate what a system does with the data after it has already crawled it." Then the short version: "Disallow bluntly means, 'Do not enter the room.'" (Quotes from the full transcript Search Engine Roundtable posted, HTTP 200 this fire.)

That matters for AI visibility because the AI companies split their crawlers by job. Block the wrong one and you're not saying "don't train on me." You're saying "don't show me."

The short version

• robots.txt is about entry, not use. It says which crawler may fetch which URLs. Google's own spec says it reads four fields: user-agent, allow, disallow, sitemap. Anything else, like a made-up "No-AI-Training" line, gets ignored by Google's parser.
• AI companies use different tokens for different jobs. OpenAI's OAI-SearchBot decides whether you can show up in ChatGPT search answers. GPTBot is the training signal. They're independent.
• Google-Extended is not a Search switch. Google says it controls Gemini training and Gemini grounding, and "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
• AI Overviews and AI Mode have their own control. Google's documentation points to a site-wide "Search generative AI control" in Search Console, not robots.txt.
• A standard "use" rule doesn't exist yet. There's an IETF draft for a Content-Usage line. It's a draft.
• Check your file before you change anything. Map each token to the product it changes. Decide on purpose.

What each token actually changes

Token in robots.txt	What the owner says it controls	Source (opened this fire)
Googlebot	Google Search, including Discover and all Search features, plus Images, Video, News	Google's common crawlers list
Google-Extended	Whether Google-crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Has no user agent of its own; it's "used in a control capacity." Doesn't affect Search inclusion or ranking	Same page
Search generative AI control (Search Console, not robots.txt)	Your site's inclusion in generative AI features on Google Search, "including AI Overviews and AI Mode." Site-wide	Google, Control what you share with Google Search
OAI-SearchBot	Whether your site can be surfaced in ChatGPT search answers. Opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI says changes take about 24 hours to apply	OpenAI, Overview of OpenAI Crawlers
GPTBot	Training. Disallowing it "indicates a site's content should not be used in training generative AI foundation models"	Same page
ChatGPT-User	Visits a person triggers inside ChatGPT. OpenAI says "robots.txt rules may not apply" and it isn't used to decide Search inclusion	Same page

Other AI companies publish their own tokens. I didn't open their pages this fire, so they aren't in the table. Check each vendor's own documentation before you write a rule for it.

The three mistakes I'd look for first

1. Blocking "all AI" and losing ChatGPT search. The common copy-paste block disallows every AI bot someone could name. If OAI-SearchBot is on that list, OpenAI says your pages won't be shown in ChatGPT search answers. If what you actually wanted was "don't train on my content," OpenAI's own page gives the split: allow OAI-SearchBot, disallow GPTBot.

2. Thinking Google-Extended pulls you out of AI Overviews. It doesn't, according to Google's own description. Google-Extended covers Gemini training and grounding in the Gemini app and Vertex AI. For AI Overviews and AI Mode, Google's documentation points to the Search generative AI control in Search Console. And blocking Googlebot to get out of AI features takes you out of Search with it.

3. Writing rules nobody reads. Google's spec lists the four fields it supports and says "other fields such as crawl-delay aren't supported." Illyes on the video: a fancy custom rule gets thrown "in the trash." There's a quieter trap in the same spec: "Only one group is valid for a particular crawler." Google's crawlers pick the group with the most specific matching user agent and ignore the rest. So if you add a User-agent: GPTBot group to allow it, that bot no longer follows the Disallow lines you wrote under User-agent: *. Copy any private-path rules into the named group too.

A five-minute check

1. Open yourdomain.com/robots.txt in a browser. If it's empty or missing, every well-behaved crawler can enter.
2. List every User-agent line. For each one, write down what it controls, from the owner's documentation (the table above covers Google and OpenAI).
3. Ask the question on purpose: do you want to appear in ChatGPT search? In Gemini answers? In AI Overviews? Those are three separate settings in three separate places.
4. If you added named-bot groups, check each one still carries the Disallow rules you need. A named group replaces the * group for that bot.
5. Remember what robots.txt won't do. It isn't security; Google's spec notes a disallowed URL can still be indexed and shown without a snippet. For real exclusion, Google points to noindex, password protection, or removing the page.

Then do the part that actually earns citations: make sure the pages you're letting in say the true answer about your business, clearly, on your own domain. Letting a crawler in doesn't make you the source it cites. It just means you're in the room.

What's coming: a usage rule, maybe

Illyes co-wrote, with Mozilla's Martin Thomson, an IETF draft that would add a Content-Usage line to robots.txt and a matching HTTP header, so a site could say "crawl me, but here's how you may use it." The draft's example is Content-Usage: train-ai=n under an Allow: /. Per the IETF Datatracker entry for draft-ietf-aipref-attach-05 (HTTP 200 this fire), it's an active working-group document, last updated August 18, 2026, intended as a Proposed Standard, and set to expire February 20, 2027 unless revised. It hasn't gone to the IESG. Nothing reads it as a rule today. Don't build on it yet.

What this is not (honesty block)

• Not a ranking claim. Nobody here says allowing a crawler gets you cited. Google says Google-Extended isn't a ranking signal. OpenAI says opting out of OAI-SearchBot keeps you out of ChatGPT search answers; it doesn't say opting in gets you cited.
• Not a full vendor list. Only Google's and OpenAI's own pages were opened this fire. Anthropic, Perplexity, Microsoft, Apple, Meta, and others publish their own tokens; check those pages directly.
• Not a statement about content already collected. Google's video says robots.txt doesn't govern use after crawling. OpenAI's page doesn't say what disallowing GPTBot does to content collected before you changed the file. We don't know.
• Not a standard for "don't train." The IETF Content-Usage rule is a draft. Google-Extended and GPTBot are each company's own product controls, not RFC 9309.
• Not data on other parsers. Google's hosts said "other or older crawlers on the web might get confused" by raw non-English characters in rule paths. They didn't name which crawlers or how often. Use percent-encoded paths to be safe.
• Not security. robots.txt asks well-behaved crawlers to stay out. It doesn't stop anyone else.
• Not a promise about user-triggered visits. OpenAI says robots.txt rules "may not apply" to ChatGPT-User.

Related New Reward Field Guides

• Cloudflare is helping ChatGPT find more pages. Found is not cited.
• Web Bot Auth is identity, not a citation
• AI search reporting is becoming an SEO signal
• AI search cited a made-up concept within a week
• AI sends a sliver of your traffic. Do the work in the right order.

Want the fast read first?

The free AI-visibility score is a quick check: email only, no credit card, a 0–100 read across the engines we track. It won't edit your robots.txt. It will tell you whether AI can see your business clearly once the door's open.