Free tool · No account
Robots.txt AI crawler checker
Paste your web address and see, crawler by crawler, which AI assistants your robots.txt lets read your site — ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and more — and which it turns away. Free, instant, no account.
How it works
How the AI crawler checker works
01
It reads your robots.txt like a crawler
Quaryan asks for your robots.txt the way a crawler does: following redirects, treating a missing file as permission to read everything and a server error as “keep out”.
02
It matches every crawler to its rules
For each of eighteen crawlers it finds the group that names it — or, if none does, the one for every crawler (*) — and applies the most specific rule to your home page, exactly as the robots.txt standard, RFC 9309, prescribes.
03
It shows you the line that decides
Allowed, allowed with exceptions, or blocked — with the line of your robots.txt that decides it, so you know exactly what to change, and whether you meant it.
The crawlers
The AI crawlers we check, and what each one does
Three kinds of crawler read your site for AI, and they are worth treating differently. Search crawlers index pages for the answers assistants give today, and link back. On-request fetchers read a page when a person asks the assistant about it. Training crawlers collect pages to train future models. Blocking the first two keeps you out of answers; blocking the third keeps your pages out of training. And two search engines — Google and Bing — feed AI answers from their own indexes.
| User-agent | Company | What it does |
|---|---|---|
| AI search and answers | ||
OAI-SearchBot |
OpenAI | Indexes pages so ChatGPT search can show and link them. Not used for training. |
Claude-SearchBot |
Anthropic | Indexes pages to improve the search results in Claude's answers. |
PerplexityBot |
Perplexity | Indexes pages for Perplexity's answers, which cite and link their sources. |
DuckAssistBot |
DuckDuckGo | Fetches pages to write DuckDuckGo's AI-assisted answers. |
| Fetching a page a person asks about | ||
ChatGPT-User |
OpenAI | Fetches a page when a ChatGPT user asks about it or shares a link. |
Claude-User |
Anthropic | Fetches a page when a Claude user asks about it. |
Perplexity-User |
Perplexity | Fetches a page when a Perplexity user asks about it. |
MistralAI-User |
Mistral AI | Fetches a page when a user of Mistral's Le Chat asks about it. |
| AI training | ||
GPTBot |
OpenAI | Collects pages to train OpenAI's models. |
ClaudeBot |
Anthropic | Collects pages to train Anthropic's Claude models. |
Google-Extended |
Not a crawler but a token: decides whether pages Google crawls may be used for its Gemini models. Does not affect Google Search. | |
Applebot-Extended |
Apple | A token: decides whether pages Applebot crawls may train Apple's AI models. Does not affect Siri or Spotlight. |
Meta-ExternalAgent |
Meta | Collects pages for Meta's AI, including training its models. |
Amazonbot |
Amazon | Amazon's crawler, used to improve its services, including training AI models. |
CCBot |
Common Crawl | Common Crawl's crawler. Its free web archive is a common source of AI training data. |
Bytespider |
ByteDance | ByteDance's crawler, used to train its AI models. |
| Search engines AI answers draw on | ||
Googlebot |
Google Search, including AI Overviews and AI Mode. | |
Bingbot |
Microsoft | Bing search, which Microsoft Copilot's answers draw on. |
Allow or block
How to allow or block AI crawlers in robots.txt
Robots.txt is a plain text file at the root of your site. Each group starts with the crawlers it is for (User-agent) and lists what they may not read (Disallow) or may (Allow). Three common set-ups:
Let every crawler in
The default, and the same as having no robots.txt at all.
User-agent: *
Allow: /Keep AI training out, stay in AI answers
Turns away the crawlers that collect pages for training, and lets in the ones that put you in answers and send visitors.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /Turn away one crawler
One group names it; everything else is unchanged.
User-agent: GPTBot
Disallow: /Free · No account · A few seconds
FAQ
Questions about AI crawlers and robots.txt
How do I check whether my robots.txt blocks AI crawlers?
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
Should I block AI crawlers?
Does blocking Google-Extended remove my site from Google Search or AI Overviews?
Does robots.txt stop AI companies from using my content?
What happens if my robots.txt returns an error?
What is llms.txt?
Is the AI crawler checker free?
Next
Crawlers can read you. Does ChatGPT name you?
Being readable is the first step. The free ChatGPT visibility check asks ChatGPT what your customers ask and shows whether it names you — and who it names instead.