# Robots.txt for AI Crawlers in 2026: Allow, Block, or Negotiate? **URL:** https://organikpi.com/blog/technical-seo/robots-txt-ai-crawlers/ **Published:** 2026-04-27 **Modified:** 2026-07-02 **Author:** Daniel Shashko > robots.txt now controls two independent levers for AI: search crawler access (which drives citations) and training crawler access (which affects model data). OpenAI publishes three bots: GPTBot (training), OAI-SearchBot (ChatGPT search citations), and ChatGPT-User (user-triggered). Anthropic added Claude-SearchBot in April 2026, joining ClaudeBot (training) and Claude-User. Google-Extended controls only Gemini Apps and Vertex AI Grounding training: AI Overviews and AI Mode run on Googlebot, not Google-Extended. Perplexity publishes PerplexityBot (search index) and Perplexity-User (user fetch). Blocking OAI-SearchBot removes ChatGPT search citations; blocking GPTBot does not. In our 153,425-citation study, 76.95% of cited URLs were outside the organic top-10, confirming AI engines draw from a broad crawl pool that requires search-bot access to enter. --- > robots.txt now controls two independent levers for AI: search crawler access (which drives citations) and training crawler access (which affects model data). OpenAI publishes three bots: GPTBot (training), OAI-SearchBot (ChatGPT search citations), and ChatGPT-User (user-triggered). Anthropic added Claude-SearchBot in April 2026, joining ClaudeBot (training) and Claude-User. Google-Extended controls only Gemini Apps and Vertex AI Grounding training: AI Overviews and AI Mode run on Googlebot, not Google-Extended. Perplexity publishes PerplexityBot (search index) and Perplexity-User (user fetch). Blocking OAI-SearchBot removes ChatGPT search citations; blocking GPTBot does not. In our 153,425-citation study, 76.95% of cited URLs were outside the organic top-10, confirming AI engines draw from a broad crawl pool that requires search-bot access to enter. Your robots.txt file now controls two entirely different things for AI: whether AI engines cite you in live answers, and whether they train their models on your content. Getting the directives wrong collapses both into one decision when they are actually independent levers. Seven major AI platforms published official crawler documentation as of mid-2026. Each one separates at least two distinct user agents: a search/retrieval bot and a training bot. The same robots.txt entry does not do both jobs equally. ## The Verified 2026 AI Crawler List Every user agent string below is taken directly from the vendor’s official documentation page. These strings get copy-pasted into production robots.txt files. ### OpenAI (source: platform.openai.com/docs/bots) OpenAI publishes three crawlers with distinct roles. **GPTBot** crawls for training data. Disallowing GPTBot signals that your content should not be used to train OpenAI’s generative foundation models. **OAI-SearchBot** crawls to surface websites in ChatGPT search results. Disallowing it removes your site from ChatGPT search answers entirely. **ChatGPT-User** is not an automatic crawler: it fires when a user explicitly asks ChatGPT to visit a page, so robots.txt rules may not apply to it. These three roles are independent of each other: you can allow OAI-SearchBot and block GPTBot simultaneously. # OpenAI: block training only, allow search User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / # ChatGPT-User: user-triggered, robots.txt may not apply User-agent: ChatGPT-User Allow: / ### Anthropic (source: support.anthropic.com) Anthropic added a third bot in April 2026, expanding from one crawler to three. **ClaudeBot** collects web content that may contribute to model training. **Claude-SearchBot** indexes content for Claude’s search results. Disabling it reduces your site’s visibility in Claude search responses. **Claude-User** fires when a Claude user asks a question that fetches a live page. Disabling Claude-User prevents Claude from retrieving your content during user queries, which can reduce your visibility in user-directed web search. The distinction matters: blocking ClaudeBot alone does not block Claude-SearchBot. # Anthropic: block training, allow search and user access User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / ### Google (source: developers.google.com/crawling) **Google-Extended** is a standalone product token that web publishers can use to manage whether their crawled content may be used to train Gemini Apps and Grounding with Google Search on Vertex AI. It does not control Google Search, AI Overviews, or AI Mode. AI Overviews and AI Mode run on the standard **Googlebot** user agent. Blocking Google-Extended stops Gemini model training. Blocking Googlebot stops everything: organic search, AI Overviews, and AI Mode visibility simultaneously. If you want to appear in AI Overviews, you must allow Googlebot, full stop. # Google: block Gemini training, keep search + AI Overviews User-agent: Google-Extended Disallow: / User-agent: Googlebot Allow: / ### Perplexity (source: docs.perplexity.ai/guides/bots) **PerplexityBot** crawls to surface and link websites in Perplexity search results. It is not used to crawl content for AI foundation models. **Perplexity-User** runs when users ask Perplexity a question that fetches a live page. It is not used for training data collection either. Blocking PerplexityBot removes you from Perplexity search results and, indirectly, from [Perplexity citations](https://organikpi.com/blog/geo-ai-search/perplexity-citation-strategy/). ## The Train vs. Search Distinction You Cannot Ignore The most expensive mistake we see in [AI crawler management](https://organikpi.com/blog/technical-seo/robots-txt-ai-crawlers/) is conflating training crawlers with search crawlers. They produce opposite outcomes when you block them. Blocking **GPTBot** tells OpenAI not to train on your content. It has zero effect on whether ChatGPT search cites you. Blocking **OAI-SearchBot** tells ChatGPT search not to index your site. It does not stop OpenAI from training on your content. If your goal is AI visibility (citations in ChatGPT answers), you need to allow OAI-SearchBot. If your goal is preventing training, you need to block GPTBot. These are different directives solving different problems. The same logic applies across every vendor. Blocking ClaudeBot does not block Claude-SearchBot. Blocking PerplexityBot does not prevent Perplexity from using content it already indexed. Blocking Google-Extended does not affect whether your page appears in an AI Overview. Each directive controls exactly one thing. In our [42,971-citation study](https://organikpi.com/blog/geo-ai-search/decoded-42971-ai-citations-google-research/) across six AI platforms, the platforms drawing the most citations all operated search crawlers separate from training infrastructure. Their citation behavior was driven by freshness, authority, and content structure. You can opt out of training and still get cited heavily. How to set robots.txt directives depending on whether your brand wants AI search citations or wants to block training crawlers. ## What Google-Extended Actually Controls (and Does Not) Google-Extended was introduced in September 2023 and updated to clarify scope. Its documented purpose is control over two specific Google AI products: Gemini Apps and Grounding with Google Search on Vertex AI. AI Overviews and AI Mode both run on Googlebot, outside Google-Extended scope entirely. The Google documentation is explicit on this point. This creates a practical problem for brands that want to “opt out of AI” on Google without giving up search: there is no clean way to do it. Blocking Google-Extended stops Gemini model training but leaves AI Overviews intact. Blocking Googlebot stops everything. There is no directive that blocks only AI Overviews while keeping organic rankings. For more on how Google AI Overviews select sources, see our [AI Overviews optimization guide](https://organikpi.com/blog/geo-ai-search/google-ai-overviews-optimization-playbook/). Brands asking us whether to block Google-Extended for competitive reasons should know what they are actually giving up: future Gemini App training inclusion and Vertex AI Grounding data. Current search rankings and AI Mode presence are unaffected. For [publishers weighing AI training data licensing](https://organikpi.com/blog/brand-authority/content-licensing-ai-training-publishers/), Google-Extended is the lever to pull. ## Crawler Reference Table VendorUser AgentPurposeBlock removes you fromOpenAIGPTBotTraining dataFuture OpenAI model trainingOpenAIOAI-SearchBotChatGPT searchChatGPT search answersOpenAIChatGPT-UserUser-triggered fetchUser-directed ChatGPT queriesAnthropicClaudeBotTraining dataFuture Claude model trainingAnthropicClaude-SearchBotClaude search indexClaude search resultsAnthropicClaude-UserUser-triggered fetchUser-directed Claude queriesGoogleGoogle-ExtendedGemini + Vertex trainingGemini Apps and Vertex AI GroundingGoogleGooglebotSearch + AI OverviewsAll Google search and AI featuresPerplexityPerplexityBotPerplexity search indexPerplexity search and citationsPerplexityPerplexity-UserUser-triggered fetchUser-directed Perplexity queries ## Our Recommended robots.txt Blocks ### For brands that want maximum AI search visibility This configuration allows all search crawlers, blocks all training crawlers, and preserves maximum citation surface across every major AI platform. We run this on our own site. # ALLOW: All AI search bots (citations and search answers) User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Googlebot Allow: / # BLOCK: Training-only crawlers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / # BLOCK: Generic scrapers (not verified as specific AI vendors) User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / ### For brands that want to block all AI crawlers This configuration blocks all known AI-specific crawlers. Note: blocking Googlebot removes you from AI Overviews and AI Mode in addition to organic search. This is an aggressive stance with real visibility cost. For most brands it is the wrong default. # BLOCK ALL AI CRAWLERS User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / # WARNING: blocking Googlebot removes you from Google Search AND AI Overviews/AI Mode # User-agent: Googlebot # Disallow: / ## Notes on Unverified Crawlers CCBot (Common Crawl) and Bytespider (ByteDance) appear in [AI crawler log file analysis](https://organikpi.com/blog/technical-seo/ai-crawler-log-file-analysis-citation-optimization/) across many sites. Neither vendor publishes a crawler documentation page equivalent to OpenAI’s, Anthropic’s, or Perplexity’s at the time of writing. We include them in the block list for training data because they are widely cited in industry discussion, but we cannot verify their full user agent strings from a primary documentation page the same way we verified the crawlers above. The Amazonbot user agent follows a similar pattern: Amazon’s crawler documentation has not reached the same level of granularity as the major AI platforms. Treat these entries as best-effort blocks rather than authoritative configurations. For a deeper look at how these crawlers behave in practice, our [AI crawler log analysis guide](https://organikpi.com/blog/technical-seo/ai-crawler-log-file-analysis-citation-optimization/) walks through reading server logs to identify who is actually crawling your site and at what frequency. ## Our Stance: We Allow AI Crawlers OrganikPI allows all AI search crawlers and selectively blocks training crawlers. Our reasoning: [generative engine optimization](https://organikpi.com/blog/geo-ai-search/what-is-geo-generative-engine-optimization/) only works if AI platforms can actually read your content. Our [GEO/AEO Tracker](https://organikpi.com/tools/geo-aeo-tracker/) monitors citation frequency across ChatGPT, Perplexity, Claude, and Gemini. In our [May 2026 study of 153,425 citations](https://organikpi.com/blog/seo-strategy/ai-mode-text-fragments-dead-153425-citations/), 76.95% of cited URLs were not in the organic top-10. AI engines pull from a broad crawl pool, and blocking their search bots simply removes you from that pool. The training question is a separate business and legal consideration. If you want to audit which AI crawlers currently index your site and what content they are reaching, our [GEO audit service](https://organikpi.com/services/geo-audit/) includes a crawler access review as part of the technical baseline. The [GEO audit checklist](https://organikpi.com/blog/geo-ai-search/geo-audit-checklist/) covers the robots.txt review as one of 50 checkpoints. The [llms.txt adoption report](https://organikpi.com/blog/distribution/llms-txt-adoption-impact/) covers a related but separate signal: the voluntary llms.txt file that some sites use to guide AI agents. Unlike robots.txt, Google has stated explicitly that it ignores llms.txt. For most brands, robots.txt remains the operative access control layer, and it needs to be configured with the train-vs-search distinction in mind. For a full view of how these technical signals connect to citation performance, our [GEO optimization service](https://organikpi.com/services/geo-optimization/) maps crawler access, content structure, and authority signals into a single engagement. The crawler layer is a prerequisite: if AI search bots cannot read you, none of the content or authority work matters. ## Frequently Asked Questions ### Does blocking GPTBot prevent ChatGPT from citing my site? No. GPTBot is the training crawler. Blocking it prevents OpenAI from using your content in future model training but has no effect on ChatGPT search citations. To control ChatGPT search visibility, you need to configure OAI-SearchBot in your robots.txt. These two directives are independent. ### Does Google-Extended control AI Overviews and AI Mode? No. Google-Extended is a standalone product token that controls whether your content is used in Gemini Apps and Grounding with Google Search on Vertex AI. AI Overviews and AI Mode run on the standard Googlebot user agent. To block AI Overviews appearances, you would need to block Googlebot, which also removes you from organic search entirely. ### What did Anthropic change in April 2026? Anthropic expanded from one public crawler (ClaudeBot) to three. ClaudeBot handles training data collection. Claude-SearchBot indexes content for Claude search results, and disabling it reduces your site's visibility in Claude search responses. Claude-User fires when a user asks Claude a question that fetches a live page. Each directive controls a different aspect of Claude visibility. ### Which robots.txt configuration maximises AI search citations? Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Claude-User. Keep Googlebot allowed (this is mandatory for AI Overviews and AI Mode). Optionally block the training-only bots (GPTBot, ClaudeBot, Google-Extended) if you want to opt out of training data use. In our 153,425-citation analysis, 76.95% of cited URLs were not in the organic top-10, meaning AI engines cite broadly from their crawl pool. ### Are CCBot and Bytespider verified AI crawler user agents? Neither CCBot (Common Crawl) nor Bytespider (ByteDance) publishes a formal crawler documentation page with verified user agent strings at the same level of detail as OpenAI, Anthropic, or Perplexity. They appear frequently in server logs and are widely blocked in robots.txt configurations, but treat them as best-effort blocks rather than verified primary-source directives.