Cloudflare AI Crawl Control: How to Block, Charge, or Allow AI Bots

Cloudflare AI Crawl Control lets you see every AI bot hitting your site and decide, per crawler, whether to allow it, block it, or charge it, and it is free on all Cloudflare sites with no configuration. That matters because AI training crawlers take far more than they return: in June 2025 Cloudflare measured a crawl-to-referral ratio of about 14 to 1 for Google, 1,700 to 1 for OpenAI, and 73,000 to 1 for Anthropic, while AI training crawler traffic rose 65% over the prior six months. The tool launched as AI Audit on September 23, 2024 and moved to general availability as AI Crawl Control on August 28, 2025. The catch for anyone doing GEO: block the wrong bot and you delete yourself from AI answers.

This is a practitioner walkthrough of the live product, not a strategy overview. For the broader picture of how autonomous AI agents reshape who gets crawled and cited, see our guide to AI agent browsing and MCP servers. Here we configure the dashboard itself, screen by screen, with the single rule that keeps you visible: block training crawlers, allow search crawlers.

What AI Crawl Control actually is

Cloudflare shipped the feature as AI Audit on September 23, 2024, free to every Cloudflare site with no setup required. It rebranded to AI Crawl Control and moved from beta to general availability on August 28, 2025. The job is simple to state. It identifies which AI operators crawl your domain, shows you what they took, and gives you a per-crawler switch to allow, block, or monetize each one.

The scale of the problem is why this exists. Cloudflare reports that GPTBot accessed 28.97% of Cloudflare-protected sites, more than any other AI bot. Defenses are thin: only about 37% of the top 10,000 domains even publish a robots.txt, and GPTBot is disallowed in just 7.8% of those. If you are not actively managing this, you are almost certainly being crawled and giving content away for nothing back. Over 1 million Cloudflare customers have already enabled the one-click block toggle Cloudflare launched in July 2024.

Step 1: Read the Overview dashboard

Open AI Crawl Control and the Overview is the first thing you see. It reports total AI crawler requests, splits them into allowed versus unsuccessful, and breaks the traffic down by operator: Google, Microsoft, ByteDance, Anthropic, Apple, and the rest. A managed robots.txt panel sits alongside the crawler grid so you can see your policy and your traffic in one place.

Read the operator breakdown before you touch a single toggle. Your goal is to identify which bots are training crawlers and which are search crawlers, because those two groups demand opposite treatment. Pair this view with your own server logs for full coverage; our guide to AI crawler log file analysis shows how to map crawls to citation wins, and the same logic feeds the dashboard you build in our AI search traffic dashboard in Looker Studio.

Step 2: Block or allow each crawler

The Security view is where the work happens. Every detected crawler gets its own row with a category, a request count, and a one-click Block toggle. You will see ClaudeBot, GPTBot, Bytespider, Applebot, OAI-SearchBot, PerplexityBot, and CCBot among others. This is the control surface for the entire strategy.

Before you flip toggles at random, learn the bot identities. Operators run separate bots for separate jobs, and Cloudflare names each one. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User as the assistant. Anthropic runs ClaudeBot for training, Claude-SearchBot for search, and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Google runs Googlebot for search, Google-Extended for AI training, and Google-CloudVertexBot. Add Bytespider from ByteDance, CCBot from Common Crawl, meta-externalagent from Meta, Applebot and Applebot-Extended, and Amazonbot.

The GEO trade-off: training versus search crawlers

Here is the single insight that should drive every toggle. Blocking a training crawler does not remove you from that company’s AI search answers. Block GPTBot and you can still be cited in ChatGPT Search through OAI-SearchBot. Block ClaudeBot and Claude-SearchBot still works. Training and search are different pipelines with different bots, so you can starve the training pipeline while staying fully visible in the answer engines that send referrals.

The smart default falls out of this directly: block the training crawlers, allow the search crawlers.

One warning before you celebrate. A robots.txt directive is voluntary, and crawlers can ignore it. Only the network-level WAF rule actually guarantees a block. We unpack the difference in robots.txt for AI crawlers in 2026, and the GEO consequences of getting this wrong are covered in how to rank in ChatGPT Search and how Perplexity actually picks sources.

The Optimization view: crawl data reframed for AI answers

As of June 2026 the live dashboard includes an Optimization tab, and it changes the framing entirely. Instead of asking only what to block, it asks how ready you are to be answered. The dashboard now surfaces an agent readiness check, an AI Search setup flow, a list of your most visited pages for AI answers, demand signals, and AI Answer volume broken down by operator.

This is the GEO mindset baked into the product: the same crawl data, reframed around visibility rather than defense. It pairs naturally with how you track AI traffic downstream. Set up attribution with our guide to AI search referral attribution in GA4, then audit your standing with the 50-point GEO audit checklist and a DIY AI brand visibility audit.

Step 3: robots.txt and the Content Signals Policy

The Signals tab shows managed robots.txt status, robots.txt availability and violations per hostname, and agent readiness indicators. Cloudflare’s managed robots.txt is built to protect SEO: it disallows Google-Extended for AI training while keeping Googlebot for search, so your rankings are untouched. A one-click “Enforce robots.txt rules” button has been available since December 10, 2024, and it turns your robots.txt directives into a deployed WAF rule, which is the only way to make a block binding.

On September 24, 2025 Cloudflare launched the Content Signals Policy, a robots.txt addition with three signals: search, ai-input, and ai-train. Each signal expresses your preference for how a crawler may use what it fetches. The absence of a signal is neutral and carries no meaning, so silence is not consent or refusal. The managed robots.txt default serves search=yes and ai-train=no, and it does not serve an ai-input signal. The syntax looks like this:

Content-Signal: search=yes, ai-train=no

That single line says crawlers may use the page for search indexing but not for AI model training. The standard is documented at the cross-vendor home for the policy, contentsignals.org, and explained in depth on the Cloudflare Content Signals Policy announcement. For the wider format picture, our deep dives on llms.txt adoption and AI training data licensing are worth a read.

Agent Readiness: a score for AI-friendliness

AI Crawl Control includes an Agent Readiness feature, surfaced on the Signals tab in the live dashboard. The “Check readiness score” button scans your robots.txt configuration, your Markdown for Agents, and your Content Signals Policy, and it is powered by isitagentready.com. Think of it as a quick health check for whether AI agents can find, parse, and use your site cleanly. It complements a structured content approach; if you are tuning pages for machine reading, see our work on BLUF writing for AI search and content chunking for AI retrieval.

Pay Per Crawl: charging bots with HTTP 402

Cloudflare announced Pay Per Crawl as a private beta on July 1, 2025, branded Content Independence Day. The model is built on the HTTP 402 Payment Required status code. For each crawler, the publisher picks one of three states: Allow, Charge, or Block. When you charge, Cloudflare acts as the Merchant of Record and handles billing between you and the AI company. Customizable 402 responses opened to all paid Cloudflare customers on August 28, 2025, and customers already send over 1 billion 402 responses per day.

Pay Per Crawl is the most aggressive lever in the panel. It turns blocking into a paywall, so instead of giving content away or refusing it outright, you set a price. It is the practical answer to those lopsided crawl-to-referral ratios. If the economics of selling versus protecting your content interest you, weigh it against your broader plan in our GEO versus SEO budget split guide.

The recommended setup, in order

1. Open the Overview and identify which operators are crawling and how often.
2. In Security, block the training crawlers: GPTBot, ClaudeBot, CCBot, Bytespider, and Google-Extended.
3. Confirm the search crawlers stay allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot.
4. Enable managed robots.txt for the search=yes, ai-train=no default, then click Enforce robots.txt rules to deploy the binding WAF rule.
5. Run the Agent Readiness check and fix whatever it flags.
6. Consider Pay Per Crawl on a paid plan if you want to charge instead of block.

AI Crawl Control is the rare control panel where the safe move and the smart move line up. You protect your bandwidth and your training rights without sacrificing a single AI citation, as long as you respect the training-versus-search split. If you would rather have someone configure all of this and run the visibility program around it, OrganikPI runs GEO for clients end to end. Start from what GEO is if the discipline is new to you.