# Cloudflare AI Crawl Control: How to Block, Charge, or Allow AI Bots **URL:** https://organikpi.com/blog/technical-seo/cloudflare-ai-crawl-control-guide/ **Published:** 2026-06-13 **Modified:** 2026-06-22 **Author:** Daniel Shashko > Cloudflare AI Crawl Control shows which AI bots crawl your site and lets you allow, block, or charge each one. It launched as AI Audit on September 23, 2024 and reached general availability on August 28, 2025. Crawl-to-referral ratios in June 2025 were about 14 to 1 for Google, 1,700 to 1 for OpenAI, and 73,000 to 1 for Anthropic, while AI training crawler traffic rose 65% in six months. GPTBot accessed 28.97% of Cloudflare sites. Block training crawlers like GPTBot and ClaudeBot, allow search crawlers like OAI-SearchBot. The Content Signals Policy default is search=yes, ai-train=no. Pay Per Crawl uses HTTP 402. --- > Cloudflare AI Crawl Control shows which AI bots crawl your site and lets you allow, block, or charge each one. It launched as AI Audit on September 23, 2024 and reached general availability on August 28, 2025. Crawl-to-referral ratios in June 2025 were about 14 to 1 for Google, 1,700 to 1 for OpenAI, and 73,000 to 1 for Anthropic, while AI training crawler traffic rose 65% in six months. GPTBot accessed 28.97% of Cloudflare sites. Block training crawlers like GPTBot and ClaudeBot, allow search crawlers like OAI-SearchBot. The Content Signals Policy default is search=yes, ai-train=no. Pay Per Crawl uses HTTP 402. Cloudflare AI Crawl Control lets you see every AI bot hitting your site and decide, per crawler, whether to allow it, block it, or charge it, and it is free on all Cloudflare sites with no configuration. That matters because AI training crawlers take far more than they return: in June 2025 Cloudflare measured a crawl-to-referral ratio of about 14 to 1 for Google, 1,700 to 1 for OpenAI, and 73,000 to 1 for Anthropic, while AI training crawler traffic rose 65% over the prior six months. The tool launched as AI Audit on September 23, 2024 and moved to general availability as AI Crawl Control on August 28, 2025. The catch for anyone doing GEO: block the wrong bot and you delete yourself from AI answers. This is a practitioner walkthrough of the live product, not a strategy overview. For the broader picture of how autonomous AI agents reshape who gets crawled and cited, see our guide to [AI agent browsing and MCP servers](https://organikpi.com/blog/technical-seo/ai-agent-browsing-mcp-server-seo/). Here we configure the dashboard itself, screen by screen, with the single rule that keeps you visible: block training crawlers, allow search crawlers. ## What AI Crawl Control actually is Cloudflare shipped the feature as AI Audit on September 23, 2024, free to every Cloudflare site with no setup required. It rebranded to AI Crawl Control and moved from beta to general availability on August 28, 2025. The job is simple to state. It identifies which AI operators crawl your domain, shows you what they took, and gives you a per-crawler switch to allow, block, or monetize each one. The scale of the problem is why this exists. Cloudflare reports that GPTBot accessed 28.97% of Cloudflare-protected sites, more than any other AI bot. Defenses are thin: only about 37% of the top 10,000 domains even publish a robots.txt, and GPTBot is disallowed in just 7.8% of those. If you are not actively managing this, you are almost certainly being crawled and giving content away for nothing back. Over 1 million Cloudflare customers have already enabled the one-click block toggle Cloudflare launched in July 2024. ## Step 1: Read the Overview dashboard Open AI Crawl Control and the Overview is the first thing you see. It reports total AI crawler requests, splits them into allowed versus unsuccessful, and breaks the traffic down by operator: Google, Microsoft, ByteDance, Anthropic, Apple, and the rest. A managed robots.txt panel sits alongside the crawler grid so you can see your policy and your traffic in one place. The AI Crawl Control Overview: total AI crawler requests, allowed versus unsuccessful, and a per-operator breakdown Read the operator breakdown before you touch a single toggle. Your goal is to identify which bots are training crawlers and which are search crawlers, because those two groups demand opposite treatment. Pair this view with your own server logs for full coverage; our guide to [AI crawler log file analysis](https://organikpi.com/blog/technical-seo/ai-crawler-log-file-analysis-citation-optimization/) shows how to map crawls to citation wins, and the same logic feeds the dashboard you build in our [AI search traffic dashboard in Looker Studio](https://organikpi.com/blog/technical-seo/ai-search-traffic-dashboard-looker-studio/). ## Step 2: Block or allow each crawler The Security view is where the work happens. Every detected crawler gets its own row with a category, a request count, and a one-click Block toggle. You will see ClaudeBot, GPTBot, Bytespider, Applebot, OAI-SearchBot, PerplexityBot, and CCBot among others. This is the control surface for the entire strategy. Every detected crawler gets its own row, category, and one-click Block toggle Before you flip toggles at random, learn the bot identities. Operators run separate bots for separate jobs, and Cloudflare names each one. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User as the assistant. Anthropic runs ClaudeBot for training, Claude-SearchBot for search, and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Google runs Googlebot for search, Google-Extended for AI training, and Google-CloudVertexBot. Add Bytespider from ByteDance, CCBot from Common Crawl, meta-externalagent from Meta, Applebot and Applebot-Extended, and Amazonbot. ## The GEO trade-off: training versus search crawlers Here is the single insight that should drive every toggle. Blocking a training crawler does not remove you from that company’s AI search answers. Block GPTBot and you can still be cited in ChatGPT Search through OAI-SearchBot. Block ClaudeBot and Claude-SearchBot still works. Training and search are different pipelines with different bots, so you can starve the training pipeline while staying fully visible in the answer engines that send referrals. The smart default falls out of this directly: block the training crawlers, allow the search crawlers. BotPurposeRecommended actionCitation impactGPTBotOpenAI trainingBlockNone on ChatGPT SearchOAI-SearchBotChatGPT searchAllowKeeps you cited in ChatGPTClaudeBotAnthropic trainingBlockNone on Claude searchClaude-SearchBotAnthropic searchAllowKeeps you cited in ClaudeGoogle-ExtendedGoogle AI trainingBlockNone on Google Search or SEOGooglebotGoogle search and SEOAllowCritical, never blockPerplexityBotPerplexity searchAllowKeeps you cited in PerplexityBytespiderByteDance trainingBlockLow referral valueCCBotCommon Crawl (training datasets)BlockFeeds many models, no direct citations One warning before you celebrate. A robots.txt directive is voluntary, and crawlers can ignore it. Only the network-level WAF rule actually guarantees a block. We unpack the difference in [robots.txt for AI crawlers in 2026](https://organikpi.com/blog/technical-seo/robots-txt-ai-crawlers/), and the GEO consequences of getting this wrong are covered in [how to rank in ChatGPT Search](https://organikpi.com/blog/geo-ai-search/how-to-rank-in-chatgpt-search/) and [how Perplexity actually picks sources](https://organikpi.com/blog/geo-ai-search/perplexity-citation-strategy/). ## The Optimization view: crawl data reframed for AI answers As of June 2026 the live dashboard includes an Optimization tab, and it changes the framing entirely. Instead of asking only what to block, it asks how ready you are to be answered. The dashboard now surfaces an agent readiness check, an AI Search setup flow, a list of your most visited pages for AI answers, demand signals, and AI Answer volume broken down by operator. The Optimization view reframes the same crawl data around AI answer readiness, not just blocking This is the GEO mindset baked into the product: the same crawl data, reframed around visibility rather than defense. It pairs naturally with how you track AI traffic downstream. Set up attribution with our guide to [AI search referral attribution in GA4](https://organikpi.com/blog/technical-seo/ga4-ai-search-referral-attribution/), then audit your standing with the [50-point GEO audit checklist](https://organikpi.com/blog/geo-ai-search/geo-audit-checklist/) and a [DIY AI brand visibility audit](https://organikpi.com/blog/seo-strategy/diy-ai-brand-visibility-audit/). ## Step 3: robots.txt and the Content Signals Policy The Signals tab shows managed robots.txt status, robots.txt availability and violations per hostname, and agent readiness indicators. Cloudflare’s managed robots.txt is built to protect SEO: it disallows Google-Extended for AI training while keeping Googlebot for search, so your rankings are untouched. A one-click “Enforce robots.txt rules” button has been available since December 10, 2024, and it turns your robots.txt directives into a deployed WAF rule, which is the only way to make a block binding. Robots.txt availability, violations, and Content Signals status, per hostname On September 24, 2025 Cloudflare launched the Content Signals Policy, a robots.txt addition with three signals: search, ai-input, and ai-train. Each signal expresses your preference for how a crawler may use what it fetches. The absence of a signal is neutral and carries no meaning, so silence is not consent or refusal. The managed robots.txt default serves search=yes and ai-train=no, and it does not serve an ai-input signal. The syntax looks like this: Content-Signal: search=yes, ai-train=no That single line says crawlers may use the page for search indexing but not for AI model training. The standard is documented at the cross-vendor home for the policy, [contentsignals.org](https://contentsignals.org/), and explained in depth on the [Cloudflare Content Signals Policy announcement](https://blog.cloudflare.com/content-signals-policy/). For the wider format picture, our deep dives on [llms.txt adoption](https://organikpi.com/blog/distribution/llms-txt-adoption-impact/) and [AI training data licensing](https://organikpi.com/blog/brand-authority/content-licensing-ai-training-publishers/) are worth a read. ## Agent Readiness: a score for AI-friendliness AI Crawl Control includes an Agent Readiness feature, surfaced on the Signals tab in the live dashboard. The “Check readiness score” button scans your robots.txt configuration, your Markdown for Agents, and your Content Signals Policy, and it is powered by isitagentready.com. Think of it as a quick health check for whether AI agents can find, parse, and use your site cleanly. It complements a structured content approach; if you are tuning pages for machine reading, see our work on [BLUF writing for AI search](https://organikpi.com/blog/content-strategy/bluf-writing-ai-search/) and [content chunking for AI retrieval](https://organikpi.com/blog/technical-seo/content-chunking-rag-seo/). ## Pay Per Crawl: charging bots with HTTP 402 Cloudflare announced Pay Per Crawl as a private beta on July 1, 2025, branded Content Independence Day. The model is built on the HTTP 402 Payment Required status code. For each crawler, the publisher picks one of three states: Allow, Charge, or Block. When you charge, Cloudflare acts as the Merchant of Record and handles billing between you and the AI company. Customizable 402 responses opened to all paid Cloudflare customers on August 28, 2025, and customers already send over 1 billion 402 responses per day. Pay Per Crawl is the most aggressive lever in the panel. It turns blocking into a paywall, so instead of giving content away or refusing it outright, you set a price. It is the practical answer to those lopsided crawl-to-referral ratios. If the economics of selling versus protecting your content interest you, weigh it against your broader plan in our [GEO versus SEO budget split](https://organikpi.com/blog/geo-ai-search/geo-vs-seo-budget-split/) guide. ## The recommended setup, in order - Open the Overview and identify which operators are crawling and how often. - In Security, block the training crawlers: GPTBot, ClaudeBot, CCBot, Bytespider, and Google-Extended. - Confirm the search crawlers stay allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. - Enable managed robots.txt for the search=yes, ai-train=no default, then click Enforce robots.txt rules to deploy the binding WAF rule. - Run the Agent Readiness check and fix whatever it flags. - Consider Pay Per Crawl on a paid plan if you want to charge instead of block. AI Crawl Control is the rare control panel where the safe move and the smart move line up. You protect your bandwidth and your training rights without sacrificing a single AI citation, as long as you respect the training-versus-search split. If you would rather have someone configure all of this and run the visibility program around it, OrganikPI runs GEO for clients end to end. Start from [what GEO is](https://organikpi.com/blog/geo-ai-search/what-is-geo-generative-engine-optimization/) if the discipline is new to you. ## Frequently Asked Questions ### Does blocking AI crawlers in Cloudflare remove my site from ChatGPT or Perplexity answers? No, as long as you block the right bots. Training and search use different crawlers. If you block GPTBot, which is OpenAI's training crawler, you can still be cited in ChatGPT Search through OAI-SearchBot. If you block ClaudeBot, Claude-SearchBot still works. Allow the search crawlers and your AI citations stay intact. ### What is the difference between GPTBot and OAI-SearchBot, and which should I block? GPTBot is OpenAI's training crawler that collects content to train models. OAI-SearchBot is the crawler behind ChatGPT search. Block GPTBot to protect your content from training, and allow OAI-SearchBot so you keep showing up in ChatGPT Search answers. ### Is Cloudflare AI Crawl Control free, or do I need a paid plan? The core tool is free on all Cloudflare sites with no configuration, just as it was when it launched as AI Audit on September 23, 2024. Customizable HTTP 402 responses for charging crawlers opened to all paid Cloudflare customers on August 28, 2025, so monetization features need a paid plan. ### How does Pay Per Crawl work? Pay Per Crawl, announced as a private beta on July 1, 2025, uses the HTTP 402 Payment Required status code. For each crawler you choose one of three states: Allow, Charge, or Block. When you charge, Cloudflare acts as the Merchant of Record and handles billing. Cloudflare customers already send over 1 billion 402 responses per day. ### Does Cloudflare's managed robots.txt hurt my Google SEO? No. The managed robots.txt disallows Google-Extended, which is Google's AI training crawler, while keeping Googlebot, which handles search and SEO, fully allowed. Your rankings are not affected because the search crawler is never blocked. ### Can AI bots ignore robots.txt, and how does Cloudflare enforce it? Yes, robots.txt is voluntary and crawlers can ignore it. Cloudflare added an Enforce robots.txt rules button on December 10, 2024 that turns your robots.txt directives into a deployed WAF rule at the network level, which is the only way to make a block binding rather than a request.