Domain & SEO
Robots.txt Checker
Fetch and parse any robots.txt, then see exactly how AI crawlers and search engines are treated. Spot blocks that keep ChatGPT, Claude and Perplexity from reading your content.
Fetches /robots.txt for the domain and checks how GPTBot, ClaudeBot, PerplexityBot, Googlebot and other crawlers are treated.
Check how crawlers see your site
Enter a domain to fetch its robots.txt and see which AI engines and search bots are allowed or blocked.
Robots.txt Checker — frequently asked questions
- Which crawlers does it check?
- Eight: GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, CCBot, Googlebot, Bingbot and Google-Extended. Each is evaluated against the rules that actually govern it, following the most specific matching user-agent group.
- What is the difference between 'Blocked' and 'Partial'?
- Blocked means the site root is disallowed, so the crawler cannot read your content at all. Partial means some paths are excluded but the site remains crawlable — which is normal and usually not a problem. Most sites that exclude /admin/ or /cart are Partial, not Blocked.
- Should I block AI crawlers?
- It depends on your goal. Blocking them protects content from being used in training and answers, but it also removes you from the corpus those engines cite — so you cannot be recommended by them. If visibility in AI answers matters to you, allow them.
- Does Google-Extended affect my search rankings?
- No. Google-Extended controls whether your content is used for Gemini and Vertex AI training and grounding. It does not affect normal Googlebot crawling or your Google Search rankings — they are separate controls.