Your firewall lets Googlebot in. Google's agent needs its own pass.
Google's new guide for letting Google-Agent through your CDN and firewall shows it isn't Googlebot, isn't governed by robots.txt, and needs access well past your homepage.
By James Brady
Chief AI Officer, Product & AI Operations
Cloud AI cites who it can see. The next step is that it acts where it can get in. And a lot of sites that did the work to let Google's crawler in are still bouncing Google's agent at the door.
On November 17, Google published a short technical guide called "Allow Google-Agent traffic." Google-Agent is, in Google's words, "the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Not a crawler indexing your pages. An agent clicking through them for a person.
Last week we wrote about bot protection that walls out crawlers. This is the other half. Your "allow verified bots" setting may be doing its job for Googlebot and still not covering the thing that's trying to fill out your booking form.
The short version
- Google-Agent isn't Googlebot. It's listed as a user-triggered fetcher, a different category from Google's common crawlers, with its own IP file:
user-triggered-agents.json. - robots.txt doesn't govern it. Google says user-triggered fetchers "ignore robots.txt rules" because a user asked for the fetch. Your CDN, firewall or bot-detection service is what lets it in or keeps it out.
- Google's allow recipe has three checks: the User-Agent contains
Google-Agent, the IP is inuser-triggered-agents.json, and, if the request is signed, the Web Bot Auth signature verifies. Then: "For verified requests, skip bot challenges." - The homepage isn't the test. Google says to cover on-site search, filters, detail pages, add-to-cart, booking, and the "underlying XHR, GraphQL, or api.* endpoints."
- It's a choice, not a ranking fix. Google doesn't say allowing Google-Agent helps rankings or AI Overview citations. It's about whether an agent acting for a customer can finish the job on your site.
What Google actually said
The guide (November 17, 2026). Google says explicitly allowing Google-Agent "in your content delivery network (CDN), web application firewall (WAF), or bot-detection service helps prevent legitimate requests from being blocked while preserving your existing security controls." Each request "includes a User-Agent HTTP request header containing 'Google-Agent'" and "comes from an IP address from user-triggered-agents.json." Some providers "provide ways to verify requests made by the Google-Agent user agent and allow these requests; refer to your provider for the exact steps." Source: Google Crawling Infrastructure, Allow Google-Agent traffic. Dated in the crawling docs changelog, November 17, 2026: "Published a technical guide on how to ensure that Google-Agent has access to your website."
The troubleshooting list. This is the part worth printing. If legitimate Google-Agent requests are still blocked, Google says to check:
- Rule priority. Your allow rule must evaluate before "broad cloud or data center ASN blocks, managed bot rules, or category-level AI agent block policies that could override a verified bot setting."
- Interactive challenges. Skip "interactive CAPTCHAs and JavaScript challenges in addition to HTTP blocks, including standalone in-page CAPTCHA widgets triggered on search or form submission."
- In-journey actions and APIs. Cover "on-site search, filtering, clicking into detail pages, add-to-cart, booking, and underlying XHR, GraphQL, or api.* endpoints."
- Shared-IP rate limits. Per-IP limits should account for verified Google-Agent ranges, "as concurrent requests from multiple users share the same egress pool."
- IP range updates. Keep rules synced with the latest
user-triggered-agents.json. - Proxy headers. Load balancers and proxies must preserve the original client IP, the User-Agent, and the signature headers.
Source: same guide.
Why robots.txt is the wrong lever. Google's verification page splits its traffic into three types. Common crawlers like Googlebot "always respect robots.txt rules for automatic crawls" and use common-crawlers.json. User-triggered fetchers "ignore robots.txt rules" and use separate files, including user-triggered-agents.json. Google-Agent is on the user-triggered list.
Sources: Verify requests from Google crawlers and fetchers; List of Google user-triggered fetchers (last updated August 19, 2026, so the agent itself isn't new; the allow guide is).
The signature piece is still an experiment. Google is "experimenting with the Web Bot Auth protocol using the https://agent.bot.goog identity," signing some Google-Agent requests under RFC 9421. Its Web Bot Auth page also says: "We don't sign every request of a particular agent. Be sure that you fall back to the established methods of bot verification." Sources: Allow Google-Agent traffic; Authenticate requests with Web Bot Auth (experimental).
Why this is a visibility problem, not just an IT ticket
Here's the sequence that's coming. A person asks an AI to find someone and get it done. The AI picks from businesses it can see and trust. Then an agent goes to that business's site to check availability, fill the form, or book.
If your firewall throws a CAPTCHA at that point, the person doesn't see your site. They see a failed task. And the agent has other businesses to try.
Look at that troubleshooting list again. Almost none of it is about the homepage. It's the search box, the filter, the booking widget, the API call behind the "Check availability" button. Those are exactly the places a lot of security setups get strictest, because that's where real bots abuse forms. A rule that's correct for spam can be wrong for a customer's agent.
So the question isn't "did we allow Google?" It's "which Google, on which pages, ahead of which other rule?"
What to do with it (about 30 minutes, with whoever runs your CDN)
- Decide first. Do you want Google's agent to be able to act on your site? For most service businesses that take bookings or quotes, I'd say yes. If you don't, that's a legitimate call. Make it on purpose.
- Ask your provider one direct question. "Does our verified-bot setting cover Google-Agent, or only Google's crawlers?" Google tells you to "refer to your provider." Get the answer in writing. Don't assume the Googlebot toggle covers it.
- If it doesn't, build the allow rule Google describes. User-Agent contains
Google-Agentplus IP inuser-triggered-agents.json, signature checked when present. Never allow on the User-Agent alone. Google's own fetcher page warns the string "can be spoofed." - Move the rule up. Put it ahead of data-center ASN blocks, managed bot rules and any "block AI agents" category switch.
- Walk the money path. Homepage, on-site search, a service or product page, the booking or quote form, and whatever API that form calls. Every one of those needs the rule, including any CAPTCHA widget that fires on submit.
- Check rate limits and IP sync. Exempt or raise per-IP limits for the verified ranges. Make sure the IP list refreshes automatically. The copy I opened this morning was stamped November 18, 2026.
- Check your logs. Look for
Google-Agentin the User-Agent and see what status codes it got on form and API URLs. A wall of 403s on the booking endpoint is your answer.
Small side note: when you check how AI Mode answers about your business, Google's new temporary threads (announced November 17) run detached from your account, so "your responses won't be informed by past preferences." It's a cleaner look than your normal logged-in session. It isn't a ranking report.
Monitoring is not execution. A log line showing Google-Agent got a 200 on your booking page doesn't book anyone. The work is a site an agent can actually use: real availability, a form that doesn't hide behind pop-ups, service pages that say what you do, where, and for whom.
What this is not (honesty block)
- Not a ranking or citation factor. The guide doesn't mention rankings, AI Overviews, or AI Mode citations. Google's AI features page still says there are "no additional technical requirements" beyond being indexed and snippet-eligible.
- No product list. Google says Google-Agent is used by "agents hosted on Google infrastructure." It doesn't name which Google products send it, or how much traffic to expect. I'm not guessing.
- No provider list. Google says some CDNs and firewalls can verify Google-Agent and points you to your provider. It doesn't say which ones do it automatically. Whether your "verified bots" setting already includes it is UNCONFIRMED until your provider says so.
- Signatures aren't guaranteed. Web Bot Auth is experimental and Google doesn't sign every request. IP plus User-Agent remains the baseline.
- Not a robots.txt fix. User-triggered fetchers ignore robots.txt, so robots.txt can't invite or block Google-Agent.
- Not risk-free. Opening form and API endpoints to an agent is a security decision. Verify properly, keep your other protections in place, and decide per endpoint.
- Not the core update. The November 2026 core update is still rolling out per the Search Status Dashboard. This guide is unrelated to it, and changing firewall rules isn't a way to recover rankings.
Related New Reward Field Guides
- Your bot protection can't tell a crawler you want from one you don't. Teach it, or AI answers stop seeing you.
- Google is signing some agent requests. That is identity, not a citation.
- AI agents are getting to your booking form. Can your analytics tell?
- robots.txt is a door, not a usage policy. Know which AI bot you're actually blocking.
- Google AI Mode Can Call Your Business: What to Fix Now
Want the fast read first?
I'm not going to pretend a scan is a strategy. It's a free check of how visible your business is across the AI engines buyers use, a 0–100 score in about two minutes, email only, no card. Get your AI-visibility score.
FAQ
Common questions
What is Google-Agent?
Google calls it the name used by agents on Google infrastructure "to navigate the web and perform actions upon user request." It's a user-triggered fetcher, not Google's search crawler.
Can I block or allow it in robots.txt?
No. Google says user-triggered fetchers ignore robots.txt. Your CDN, firewall or bot-detection rules decide.
If I already allow Googlebot, am I covered?
Maybe not. Google-Agent uses a different IP file than Google's common crawlers. Ask your provider whether its verified-bot setting includes Google-Agent.
Will allowing Google-Agent get me into AI Overviews?
Google doesn't say that, and I won't either. Being cited depends on being indexed with pages worth citing. Allowing the agent is about whether it can finish a task once it gets to you.
Perplexity
Grok