AI Summary
Your robots.txt file now controls two entirely different things for AI: whether AI engines cite you in live answers, and whether they train their models on your content. Getting the directives wrong collapses both into one decision when they are actually independent levers.
Seven major AI platforms published official crawler documentation as of mid-2026. Each one separates at least two distinct user agents: a search/retrieval bot and a training bot. The same robots.txt entry does not do both jobs equally.
The Verified 2026 AI Crawler List
Every user agent string below is taken directly from the vendor’s official documentation page. These strings get copy-pasted into production robots.txt files.
OpenAI (source: platform.openai.com/docs/bots)
OpenAI publishes three crawlers with distinct roles. GPTBot crawls for training data. Disallowing GPTBot signals that your content should not be used to train OpenAI’s generative foundation models. OAI-SearchBot crawls to surface websites in ChatGPT search results. Disallowing it removes your site from ChatGPT search answers entirely. ChatGPT-User is not an automatic crawler: it fires when a user explicitly asks ChatGPT to visit a page, so robots.txt rules may not apply to it. These three roles are independent of each other: you can allow OAI-SearchBot and block GPTBot simultaneously.
# OpenAI: block training only, allow search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
# ChatGPT-User: user-triggered, robots.txt may not apply
User-agent: ChatGPT-User
Allow: /Anthropic (source: support.anthropic.com)
Anthropic added a third bot in April 2026, expanding from one crawler to three. ClaudeBot collects web content that may contribute to model training. Claude-SearchBot indexes content for Claude’s search results. Disabling it reduces your site’s visibility in Claude search responses. Claude-User fires when a Claude user asks a question that fetches a live page. Disabling Claude-User prevents Claude from retrieving your content during user queries, which can reduce your visibility in user-directed web search. The distinction matters: blocking ClaudeBot alone does not block Claude-SearchBot.
# Anthropic: block training, allow search and user access
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Google (source: developers.google.com/crawling)
Google-Extended is a standalone product token that web publishers can use to manage whether their crawled content may be used to train Gemini Apps and Grounding with Google Search on Vertex AI. It does not control Google Search, AI Overviews, or AI Mode. AI Overviews and AI Mode run on the standard Googlebot user agent. Blocking Google-Extended stops Gemini model training. Blocking Googlebot stops everything: organic search, AI Overviews, and AI Mode visibility simultaneously. If you want to appear in AI Overviews, you must allow Googlebot, full stop.
# Google: block Gemini training, keep search + AI Overviews
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /Perplexity (source: docs.perplexity.ai/guides/bots)
PerplexityBot crawls to surface and link websites in Perplexity search results. It is not used to crawl content for AI foundation models. Perplexity-User runs when users ask Perplexity a question that fetches a live page. It is not used for training data collection either. Blocking PerplexityBot removes you from Perplexity search results and, indirectly, from Perplexity citations.
The Train vs. Search Distinction You Cannot Ignore
The most expensive mistake we see in AI crawler management is conflating training crawlers with search crawlers. They produce opposite outcomes when you block them.
Blocking GPTBot tells OpenAI not to train on your content. It has zero effect on whether ChatGPT search cites you. Blocking OAI-SearchBot tells ChatGPT search not to index your site. It does not stop OpenAI from training on your content. If your goal is AI visibility (citations in ChatGPT answers), you need to allow OAI-SearchBot. If your goal is preventing training, you need to block GPTBot. These are different directives solving different problems.
The same logic applies across every vendor. Blocking ClaudeBot does not block Claude-SearchBot. Blocking PerplexityBot does not prevent Perplexity from using content it already indexed. Blocking Google-Extended does not affect whether your page appears in an AI Overview. Each directive controls exactly one thing.
In our 42,971-citation study across six AI platforms, the platforms drawing the most citations all operated search crawlers separate from training infrastructure. Their citation behavior was driven by freshness, authority, and content structure. You can opt out of training and still get cited heavily.
What Google-Extended Actually Controls (and Does Not)
Google-Extended was introduced in September 2023 and updated to clarify scope. Its documented purpose is control over two specific Google AI products: Gemini Apps and Grounding with Google Search on Vertex AI. AI Overviews and AI Mode both run on Googlebot, outside Google-Extended scope entirely. The Google documentation is explicit on this point.
This creates a practical problem for brands that want to “opt out of AI” on Google without giving up search: there is no clean way to do it. Blocking Google-Extended stops Gemini model training but leaves AI Overviews intact. Blocking Googlebot stops everything. There is no directive that blocks only AI Overviews while keeping organic rankings. For more on how Google AI Overviews select sources, see our AI Overviews optimization guide.
Brands asking us whether to block Google-Extended for competitive reasons should know what they are actually giving up: future Gemini App training inclusion and Vertex AI Grounding data. Current search rankings and AI Mode presence are unaffected. For publishers weighing AI training data licensing, Google-Extended is the lever to pull.
Crawler Reference Table
| Vendor | User Agent | Purpose | Block removes you from |
|---|---|---|---|
| OpenAI | GPTBot | Training data | Future OpenAI model training |
| OpenAI | OAI-SearchBot | ChatGPT search | ChatGPT search answers |
| OpenAI | ChatGPT-User | User-triggered fetch | User-directed ChatGPT queries |
| Anthropic | ClaudeBot | Training data | Future Claude model training |
| Anthropic | Claude-SearchBot | Claude search index | Claude search results |
| Anthropic | Claude-User | User-triggered fetch | User-directed Claude queries |
| Google-Extended | Gemini + Vertex training | Gemini Apps and Vertex AI Grounding | |
| Googlebot | Search + AI Overviews | All Google search and AI features | |
| Perplexity | PerplexityBot | Perplexity search index | Perplexity search and citations |
| Perplexity | Perplexity-User | User-triggered fetch | User-directed Perplexity queries |
Our Recommended robots.txt Blocks
For brands that want maximum AI search visibility
This configuration allows all search crawlers, blocks all training crawlers, and preserves maximum citation surface across every major AI platform. We run this on our own site.
# ALLOW: All AI search bots (citations and search answers)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Googlebot
Allow: /
# BLOCK: Training-only crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# BLOCK: Generic scrapers (not verified as specific AI vendors)
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /For brands that want to block all AI crawlers
This configuration blocks all known AI-specific crawlers. Note: blocking Googlebot removes you from AI Overviews and AI Mode in addition to organic search. This is an aggressive stance with real visibility cost. For most brands it is the wrong default.
# BLOCK ALL AI CRAWLERS
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# WARNING: blocking Googlebot removes you from Google Search AND AI Overviews/AI Mode
# User-agent: Googlebot
# Disallow: /Notes on Unverified Crawlers
CCBot (Common Crawl) and Bytespider (ByteDance) appear in AI crawler log file analysis across many sites. Neither vendor publishes a crawler documentation page equivalent to OpenAI’s, Anthropic’s, or Perplexity’s at the time of writing. We include them in the block list for training data because they are widely cited in industry discussion, but we cannot verify their full user agent strings from a primary documentation page the same way we verified the crawlers above. The Amazonbot user agent follows a similar pattern: Amazon’s crawler documentation has not reached the same level of granularity as the major AI platforms. Treat these entries as best-effort blocks rather than authoritative configurations.
For a deeper look at how these crawlers behave in practice, our AI crawler log analysis guide walks through reading server logs to identify who is actually crawling your site and at what frequency.
Our Stance: We Allow AI Crawlers
OrganikPI allows all AI search crawlers and selectively blocks training crawlers. Our reasoning: generative engine optimization only works if AI platforms can actually read your content. Our GEO/AEO Tracker monitors citation frequency across ChatGPT, Perplexity, Claude, and Gemini. In our May 2026 study of 153,425 citations, 76.95% of cited URLs were not in the organic top-10. AI engines pull from a broad crawl pool, and blocking their search bots simply removes you from that pool. The training question is a separate business and legal consideration.
If you want to audit which AI crawlers currently index your site and what content they are reaching, our GEO audit service includes a crawler access review as part of the technical baseline. The GEO audit checklist covers the robots.txt review as one of 50 checkpoints.
The llms.txt adoption report covers a related but separate signal: the voluntary llms.txt file that some sites use to guide AI agents. Unlike robots.txt, Google has stated explicitly that it ignores llms.txt. For most brands, robots.txt remains the operative access control layer, and it needs to be configured with the train-vs-search distinction in mind.
For a full view of how these technical signals connect to citation performance, our GEO optimization service maps crawler access, content structure, and authority signals into a single engagement. The crawler layer is a prerequisite: if AI search bots cannot read you, none of the content or authority work matters.