Free tool
AI crawler access checker
An AI crawler access checker reads a site's robots.txt and reports whether named crawlers may fetch a path. It checks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot and Applebot-Extended.
How to use it
- Enter a public domain, such as example.com. The checker requests https://that-domain/robots.txt.
- Add a path if you want the verdict for one URL, such as /blog, instead of the homepage.
- Read the line that decided each crawler. A named group overrides the wildcard group.
Why this matters for AI search
An answer engine can cite a page only if its crawler is allowed to fetch it. Training crawlers and search crawlers are separate controls.
OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User separately. Blocking GPTBot does not, by itself, block ChatGPT search.
Google-Extended is not the switch for Google Search or AI Overviews. Our answer-engine guide quotes Google on that point: Google-Extended does not control inclusion in Search, and AI Overviews are fetched by Googlebot.
robots.txt is a request to well-behaved crawlers. This check shows the rule you published. It does not prove that a crawler visited.
Related services
Free AI Visibility Report
These tools check files and page structure. The free AI Visibility Report shows whether answer engines cite and recommend your brand. It is delivered within 1 business day.
Get the free AI Visibility ReportQuestions
What is an AI crawler access checker?
It fetches robots.txt and applies the published grouping rules: the most specific user-agent group wins, the longest matching path wins, and Allow wins a tie. The result is a verdict per crawler, with the line that decided it.
Does blocking GPTBot remove a site from ChatGPT?
No. OpenAI's crawler docs treat GPTBot as the training crawler and OAI-SearchBot as the search crawler. Each group in robots.txt is independent. Check both rows before you change a file.
Does Google-Extended control AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews are fetched by Googlebot, which this checker does not list because it is a search crawler, not one of the nine agents above.
What if there is no robots.txt?
A missing file does not block crawlers. It also does not mean they will visit. The checker says the file was not found and does not invent an allow rule.