Surface marks show where New Reward checks visibility. They are not partner badges or outcome guarantees.

Back to Field guide
Field guide
Published |6 min read

Your bot protection can't tell a crawler you want from one you don't. Teach it, or AI answers stop seeing you.

Google says blanket bot protection without verified-crawler exemptions makes pages drop out of Search. OpenAI says to allow its published IP ranges. Here's how to check your firewall, fix it, and prove it worked.

James Brady portrait

By James Brady

Chief AI Officer, Product & AI Operations

GEOTechnical SEOCrawlingAI visibility

Cloud AI cites who it can see. A lot of businesses are making themselves harder to see without knowing it, and the culprit isn't robots.txt. It's a toggle in the CDN or firewall dashboard.

On November 11, Google's Search Central team published a post about moving a site onto a CDN. Most of it is about cache warm-up and crawl rate. The part every business owner should read is the section titled "You shall not crawl! When blanket bot protection goes too far."

The short version

  • A bot challenge is a wall for crawlers. Google says that when a CDN or Web Application Firewall (WAF) intercepts automated requests with a challenge like CAPTCHA, "Google's crawlers fail the test."
  • Google treats a walled site as broken. In Google's words, "your website appears deliberately restricted or broken. If left in place, our crawlers will back off, and your pages ... will begin to drop out of Search."
  • OpenAI asks for the same thing at the network layer. Its crawler page recommends allowing OAI-SearchBot in robots.txt "and allowing requests from our published IP ranges." Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."
  • The fix is an exemption, not an off switch. Allow verified bots, target the bots you don't want by name, and use a short 429 or 503 if your server needs a breather.
  • Then prove it. Check your logs for verified crawler hits and for visitors arriving with a Google Referer. Google documented exactly what that header looks like on November 12.

What Google actually said

The post is written for sites moving to a CDN, but the bot-protection advice applies to anyone with a firewall in front of their site.

Google explains that blanket security rules "treat all automated traffic the same, including the traffic from crawlers that you care about." Crawlers that hit a challenge page end up "stuck on an error page that's not useful for Search, or other products relying on crawled data."

Its recommended fix, in order:

  1. "If you need to block bots, allow verified bots in your CDN and firewall settings." Google notes most major CDNs and WAFs offer a built-in toggle or managed rule for known or verified bots, and that its crawlers can be identified through Google's published IP ranges. "Or better yet, configure your robots.txt file or use rules to target specific unwanted bots rather than blocking all automated clients."
  2. Pre-warm your caches before sending all traffic through a new CDN.
  3. If crawl load is a real problem, return 429 (Too Many Requests) or 503 (Service Unavailable) "for a short window (a few hours, or 1-2 days at most)."

The recap line is the one to tape to the monitor: "Don't let blanket WAF or CAPTCHA rules block Google's crawlers with 403 walls or challenge pages." Source: Google Search Central, Moving to a CDN without overwhelming your origin (or blocking Googlebot) (Nov 11, 2026).

Why this is an AI visibility problem, not just an SEO one

Robots.txt is a door with a sign on it. A bot challenge is a locked door with no sign. Crawlers that honor your robots.txt can still be turned away by your firewall, and nothing in robots.txt will tell you that happened. Related: our field guide on robots.txt as access control

Google's own phrase, "other products relying on crawled data," is broader than the blue links. And OpenAI is explicit that ChatGPT's search answers depend on OAI-SearchBot getting through. Its crawler page lists published IP files for OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot, and says each robots.txt setting is independent. You can allow search and still decline training. A firewall rule that blocks everything automated makes that choice for you, and it makes the wrong one. Source: OpenAI, Overview of OpenAI Crawlers.

If AI answers keep naming a competitor for questions your site answers well, a bot wall is one of the first things worth ruling out. It's quick to check and it's not glamorous, which is why it gets missed. Related: our field guide on AI crawlers as an SEO signal

What to do this week

  1. Find out what's in front of your site. Ask whoever runs hosting which CDN or firewall sits in front of it and which bot settings are on. "We turned on bot protection after a traffic spike" is the answer to listen for.
  2. Turn on the verified-bot allowance. Most major CDNs and WAFs have one, per Google. Use it instead of switching protection off.
  3. Name the crawlers you want. At minimum, Google's common crawlers (verify with Google's published IP ranges or a reverse-then-forward DNS lookup) and OpenAI's OAI-SearchBot (published IP file at openai.com/searchbot.json). Decide on purpose about training bots like GPTBot.
  4. Block by name, not by category. If a specific scraper is the problem, write a rule for that scraper. Don't punish every automated visitor for it.
  5. Use 429 or 503 for load, never a challenge page. Keep it short. Google says a few hours, or 1-2 days at most.
  6. Give changes time. OpenAI says robots.txt changes can take about 24 hours to reach its search systems. Google's crawlers back off from sites that look broken and ramp back up as responses get healthy.

How to prove it worked

Monitoring is not execution, but execution without proof is a guess.

  • Crawler side: look in your server or CDN logs for requests from verified Google and OpenAI IP ranges landing on real pages with 200 responses, not challenge pages. Google's verification doc walks through the DNS check.
  • Visitor side: Google's new doc explains that clicks from Google Search arrive with a Referer of https://www.google.com/ (or a local Google domain), or an android-app://com.google.android.googlequicksearchbox/... value from the Google app on Android. The header carries only the origin, never the query, and it's sometimes blank for privacy reasons.
  • Don't expect a perfect match. Google says Referer-based counts often won't line up 1:1 with Search Console clicks, because of privacy settings, redirects, and different measurement methods.

Sources: Google, Verify requests from Google crawlers and fetchers and Google, Identify Google Search traffic in your web analytics or server logs (added Nov 12, 2026). Related: our field guide on measuring ChatGPT visits without UTMs

What this is not (honesty block)

  • Not a new rule. Google didn't announce a policy change. It explained how its crawlers react to challenges and errors, in a post about CDN moves.
  • Not a statement about AI Overviews or AI Mode by name. Google's post says "Search, or other products relying on crawled data." It doesn't list which products. Connecting it to AI answers is our reading, backed by OpenAI's explicit statement for ChatGPT search.
  • The Referer doc can't separate AI clicks. Google's new page doesn't mention AI Overviews or AI Mode, and the header only carries the origin. Don't build a report that claims to split AI-answer clicks from regular result clicks using the Referer alone.
  • No timelines for drop-out or recovery. Google says crawlers "back off" and pages "begin to drop out." It doesn't say how fast, and neither will we.
  • Not every bot should get through. Blocking scrapers is reasonable. The point is to block on purpose, by name, with the crawlers you want exempted.
  • Not vendor-specific advice. We haven't tested any particular CDN's bot settings for this piece. Check your provider's own docs for the exact toggle.

Being findable starts below the content. If the crawler can't load the page, the best page on the topic doesn't exist to the answer.

Want to see whether AI answers name you for the questions you care about? Start at /scan or read how we approach SEO, AEO, and GEO together.

James Brady portrait

James Brady

Chief AI Officer, Product & AI Operations

James Brady writes from the operating-system side of New Reward: product direction, AI workflows, verification, and the proof boundaries that keep visibility work honest.

Start here

Ready to turn this into a next step?

Book the appointment if you want the live walkthrough. Run the score if you want the fast read first.

Where this fits

This is part of how New Reward improves search and AI visibility. See the SEO, AEO, and GEO offering or read how search and AI visibility work together.

Prefer New Reward in Google

Keep reading

More from the Field Guide