Playbook · 15 min read

The AI Crawler Access Playbook

Configuring robots.txt is not the same as knowing whether an AI crawler can read your site. This is the reference for the difference: every major crawler, the one mistake that hands them your admin panel, and how to check the whole chain rather than trust it.

What actually happens when an AI crawler reaches your site

Every major AI engine runs at least two separate bots: one that trains future models on your content, and one that fetches pages at query time to answer a question and cite it.

Treating "the AI crawler" as one bot is the first mistake most robots.txt files make. OpenAI, Anthropic and Perplexity all split the job in two, and Google splits it differently again.

OpenAI runs GPTBot to collect training data, and a separate OAI-SearchBot to build the index behind ChatGPT's search feature, the one that produces citations. A third bot, ChatGPT-User, fetches a single page only when a person's live prompt needs it, according to OpenAI's own developer documentation.

Anthropic's structure mirrors this. ClaudeBot collects training data, Claude-SearchBot indexes content for Claude's web answers, and Claude-User fetches a page during an active user session. Anthropic's Claude Help Center confirms all three respect robots.txt independently, including Claude-User.

Perplexity runs the same pattern with PerplexityBot and Perplexity-User. Google is the outlier: Googlebot handles search indexing as it always has, and a separate token, Google-Extended, governs whether that same crawled content can be used for Gemini app training and grounding.

The practical consequence: blocking a training crawler is a data policy decision. It has no effect on whether the search or live-fetch bot from the same company can still read and cite your site. See what generative engine optimisation actually covers for how this fits the wider discipline.

Every major AI crawler, and what blocking it actually stops

The bots share a shape, a training crawler, a search or grounding crawler, and sometimes a live user fetch, each independently controllable in robots.txt.

Two rows are worth pausing on. First, blocking Googlebot is the one entry on this table that costs you conventional search visibility too, so it is rarely the right lever. Second, Google-Extended is commonly assumed to control AI Overviews. It does not: Google's own documentation states that AI Overviews and AI Mode are Search features governed by standard indexing and snippet controls, not by Google-Extended, which covers only Gemini app and Vertex AI uses.

The full grouping mechanics behind this table come from RFC 9309, the IETF standard for the Robots Exclusion Protocol, and Google's own implementation notes, both listed in the sources below.

The single robots.txt mistake that undoes every block you set

A named User-agent group replaces the wildcard group entirely for that crawler. It does not inherit any of the wildcard's Disallow lines, so a bare 'Allow: /' hands over everything.

RFC 9309 and Google's own robots.txt documentation agree on this mechanic: a crawler matches exactly one group in the file, the most specific one addressed to it, and ignores every other group completely. Global and named groups are never combined.

That means a common pattern is a silent trap. A site writes a careful wildcard group with five Disallow lines protecting /admin, /api/ and a few gated paths. Then, wanting to welcome AI crawlers explicitly, it adds User-agent: GPTBot with just Allow: /. GPTBot now ignores the wildcard group entirely and has full run of every path the wildcard was trying to protect.

We found exactly this defect on our own site during an August 2026 robots.txt review, and flagged the mechanism briefly in the AI Search Visibility Field Manual. This is the fuller fix. Our file had grown to 22 separate crawler groups, one wildcard plus 21 named search and AI bots, and only the wildcard group carried the five protective Disallow lines. Every named group underneath it was a bare welcome mat.

The fix has one rule: repeat every Disallow line in every named group, identically, or do not create named groups at all and rely solely on the wildcard. There is no partial version of this fix. A group missing even one Disallow line is a group with a gap.

How to verify crawler access actually works, not just how you configured it

A correct robots.txt proves what you intended, not what happened. Verification means checking what a crawler actually received, in logs, not what a file says it should receive.

Three failure modes sit underneath a perfectly written robots.txt, and none of them show up by reading the file.

The first is rendering. If a page's content only appears after client-side JavaScript runs, a crawler that does not execute that script sees an empty shell regardless of what robots.txt allows. We cover the specific header that gives this away in the header AI crawlers check first.

The second is infrastructure sitting in front of the origin: a CDN rule, a bot-management product, or a WAF can block a crawler's user-agent before the request ever reaches robots.txt, or reaches your server at all. Robots.txt is a request; these systems can simply refuse to grant it.

The third is a soft block: a 200 response wrapping a cookie banner, a paywall interstitial or a CAPTCHA instead of the article itself. The crawler gets a page, just not the one you meant it to read.

The check for all three is the same: pull server or edge logs filtered to the known AI crawler user-agent strings from the table above, and confirm the response codes and response sizes look like the real page, not a redirect, a 403, or a near-empty shell. Our own step-by-step version of this check is in the AI search visibility audit checklist.

How to measure AI citations when there is no Search Console

There is no first-party reporting surface for AI citations yet. The practical stack is three layers: referral traffic, crawler hits, and direct prompt sampling, none of which alone is complete.

AI referral traffic and AI citation are two different things, and conflating them is the most common measurement mistake. Referral traffic is a real person who clicked a link inside ChatGPT, Perplexity or Gemini and landed on your site. Citation is an AI answer mentioning or drawing on your content, whether or not anyone ever clicks through.

The first layer catches referral traffic: a segment or channel group in your analytics tool that isolates known AI referrer domains, so it stops being invisible inside general referral traffic. This tells you what converts, which matters because AI referral traffic converts at a materially higher rate than Google search traffic in our own client data.

The second layer catches crawler activity: server or edge log hits from the user-agents in the crawler table, tracked over time. This tells you whether the search and live-fetch bots are actually reading new content, independent of whether anyone has clicked through yet.

The third layer is the only one that measures citation directly: running a fixed set of real prompts against each engine on a schedule and recording whether, and how, your brand or pages appear. This is manual or semi-automated today because no engine publishes a citation report. Rank-based tools alone increasingly miss this: Ahrefs' 2026 analysis of AI Overview citations across 863,000 keywords found only 38% of cited URLs still ranked in the top 10 organically, down from 76% in mid-2025, meaning most citations no longer correlate with the ranking position most teams already track. See why the name an AI recommends now functions like a front page for what that shift is worth commercially.

Where citations concentrate, and the architecture decision it forces

Citation volume concentrates in structured, product-style pages over narrative blog posts, which raises the stakes on any page type that also happens to be client rendered.

A 2026 study by Nobori.ai tracking 50,431 citations across six AI engines over 13 weeks found that product-style pages, vendor profiles, comparison pages and reference or methodology documentation, captured 76% of citations while blog posts captured 24%, despite blogs making up roughly 40% of the pages in the sample. Pages built to answer a specific question in a structured block outperformed pages built to tell a story.

This is not an argument against blog content; the answer post and signal formats on this site exist precisely because freshness and a defensible position are levers competitors cannot copy cheaply, as set out in why SEO is the floor GEO stands on. It is an argument about where the architecture risk concentrates.

If citation value clusters in product, comparison and documentation pages, and those are also the pages most likely to be built as client-rendered single-page app views for a smoother user experience, the highest-value pages on a site are also the most likely to be invisible to a crawler that does not execute JavaScript. That is a genuine tension between two teams' incentives, not a simple oversight, and it is why crawlability has to be checked page type by page type rather than once for the whole domain. It also explains why a static file like llms.txt cannot substitute for fixing it: as we found, llms.txt shows no measurable citation benefit on its own, because it changes nothing about whether the crawler can read the page it points to.

A 30 minute AI crawler access audit you can run this week

Six checks, in order: read robots.txt for the wildcard trap, spot-check response codes by user-agent, confirm rendering, set up a referral segment, check crawler log volume, and run a prompt sample.

1. Open robots.txt and confirm every named crawler group repeats every Disallow line from the wildcard group. A group with fewer Disallow lines than the wildcard is the defect from this guide's third section.
2. Fetch your three highest-value pages with a request carrying each major crawler's user-agent string and confirm a 200 response with the full page, not a redirect, a 403, or a CDN challenge page.
3. View those same three pages with JavaScript disabled. If the core content disappears, a non-rendering crawler sees what you just saw.
4. Add or confirm a referral segment for known AI domains in your analytics tool, so AI referral traffic stops being buried inside general referral traffic.
5. Pull the last seven days of server or edge logs filtered to the crawler user-agents in this guide's reference table, and note which ones show zero hits. Zero hits on a bot that should be active is itself a finding.
6. Run five real prompts a genuine buyer would ask against two or three AI engines and record whether your site is mentioned, cited, or absent.

This is the mechanical half of visibility. For the fuller picture, including the six levers that determine whether being crawlable actually earns a citation, see the AI Search Visibility Field Manual and how to get cited by ChatGPT, Perplexity and Google AI.

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content to train future OpenAI models, while OAI-SearchBot is the separate crawler that builds the index behind ChatGPT's search feature and produces citations. Blocking GPTBot stops training use only; it has no effect on whether OAI-SearchBot can still crawl and cite the site.

Does Google-Extended control whether my site appears in AI Overviews?

No. Google's own documentation states that Google-Extended governs only Gemini app training and Vertex AI grounding. AI Overviews and AI Mode are features within Google Search and are controlled by the same indexing and snippet directives, such as noindex and max-snippet, that govern regular search results.

Why would an AI crawler reach my admin pages if I have a wildcard Disallow rule?

Under RFC 9309, a named User-agent group such as 'User-agent: GPTBot' completely replaces the wildcard group for that crawler rather than inheriting its rules. If the named group only contains 'Allow: /' without repeating the wildcard's Disallow lines, that crawler has full access to every path the wildcard tried to protect.

How can I check whether AI crawlers can actually read my site, not just whether robots.txt allows it?

Pull server or edge logs filtered to known AI crawler user-agents and confirm the response codes and page sizes match the real page rather than a redirect, a blocked response, or a near-empty shell caused by client-side rendering the crawler cannot execute. A permissive robots.txt file does not guarantee any of this.

How do I measure AI citations when there is no Search Console equivalent?

Combine three layers: an analytics segment isolating AI referral traffic, server log tracking of AI crawler hits over time, and periodic manual or semi-automated sampling of real prompts against each engine to record whether your content is mentioned or cited. No single layer is sufficient on its own.

Which pages get cited most by AI search engines?

A 2026 study by Nobori.ai of 50,431 citations across six AI engines found product-style pages, including vendor profiles, comparison pages and reference documentation, captured 76% of citations against 24% for blog posts, despite blogs making up around 40% of the pages tracked. Structured, question-answering content outperformed narrative content.