robots.txt Checker
See which AI crawlers your robots.txt lets in and which ones it blocks.
robots.txt
Checker
FREE
One blocked user agent can keep your site out of ChatGPT, Claude and Perplexity answers. Enter a domain and see every major AI crawler in one view.
TOOL.
FREE
NO SIGN UP
GEO READY
Check a domain
Domain*
Domain only, no path. The check reads robots.txt and llms.txt at the root.
AI crawler access
GPTBot
/
ChatGPT-User
/
OAI-SearchBot
/
ClaudeBot
/
Claude-User
/
Claude-SearchBot
/
anthropic-ai
/
PerplexityBot
/
Perplexity-User
/
Google-Extended
/
Applebot-Extended
/
Bytespider
/
CCBot
/
Amazonbot
/
meta-externalagent
/
llms.txt present
/
Allowed, Blocked or Not mentioned, read from the robots.txt of the domain. Not mentioned means the crawler is allowed by default.
FAQ
Common questions
Which crawlers belong to which AI company?
+
OpenAI runs GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic runs ClaudeBot, Claude-SearchBot, Claude-User and the older anthropic-ai token. Perplexity runs PerplexityBot and Perplexity-User. Google-Extended belongs to Google, Applebot-Extended to Apple, meta-externalagent to Meta, Amazonbot to Amazon, Bytespider to ByteDance and CCBot to Common Crawl.
What does Not mentioned mean?
+
The robots.txt has no rule for that crawler by name, so it follows the rule for all user agents. On most sites that means it is allowed.
Does blocking Google-Extended remove me from Google?
+
No. Google-Extended only covers use of your content for Gemini models. Search, including AI Overviews, runs on Googlebot.
Should I block training crawlers and keep search crawlers?
+
You can. Search and user crawlers like OAI-SearchBot and Claude-User decide whether you appear and get cited in answers. Training crawlers like GPTBot and ClaudeBot decide whether your content goes into future models.
book a meeting
Talk to an Expert
"craze is an absolute 10 out of 10 on ease to work with.”
Matt Van Itallie
CEO, Sema
STRAIGHT FROM ADS MANAGER
Ads that pull their weight.
%20Ads%20Manager%20-%20Manage%20ads%20-%20Campaigns%201.webp)
%20Ads%20Manager%20-%20Manage%20ads%20-%20Campaigns%201.webp)
SEO and GEO
Find your missing
piece
Technical optimization, content strategy and authority building, so you stay visible across Google and the AI engines that now answer first.
