Enter any domain to see which AI crawlers its robots.txt allows or blocks. The tool reads the live robots.txt, checks it against 16 known AI crawlers from OpenAI, Anthropic, Google, Perplexity and others, and tells you exactly who can reach the site. It reads only the public robots.txt and stores nothing.
Reads the site's robots.txt and shows which AI crawlers can access it. Nothing is stored.
Which crawlers it checks
The checker covers the crawlers that matter for AI visibility, grouped by what they do:
- Training crawlers that collect content to train models: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, cohere-ai and Meta-ExternalAgent.
- Search and answer crawlers that index pages so an assistant can cite them: OAI-SearchBot, PerplexityBot, Claude-SearchBot and Amazonbot.
- Live fetchers that pull a page when a user asks a question in real time: ChatGPT-User, Perplexity-User and Claude-User.
Should you block AI crawlers?
It depends on the crawler. Blocking training crawlers like GPTBot keeps your content out of future model training, which some publishers want. Blocking the search and live crawlers is different: it removes you from the answers those assistants give, so most brands that want to be found and cited should leave them open. Our guide to robots.txt and AI crawlers walks through the trade-offs, and the wider strategy sits in answer engine optimization and AEO versus SEO.
One point people get wrong: blocking Google-Extended controls only Gemini training and grounding. It does not remove you from Google Search or AI Overviews, which follow Googlebot.
Free, private, no signup
The checker reads only the public robots.txt at the domain you enter and keeps nothing. There is no login and no cost. If you would rather have your AI visibility handled end to end, see our managed AEO service, or run a full AI visibility audit.