Free tool · No account
Crawler check
Paste your web address and see, crawler by crawler, which AI assistants your robots.txt lets read your site, from ChatGPT and Claude to Perplexity and Gemini, and which it turns away. It checks your llms.txt and meta description too. Free, no account.
How it works
How the crawler check works
01
It reads your robots.txt like a crawler
Quaryan asks for your robots.txt the way a crawler does: following redirects, treating a missing file as permission to read everything and a server error as “keep out”.
02
It matches every crawler to its rules
For each of eighteen crawlers it finds the group that names it — or, if none does, the one for every crawler (*) — and applies the most specific rule to your home page, exactly as the robots.txt standard, RFC 9309, prescribes.
03
It shows you the line that decides
Allowed, allowed with exceptions, or blocked — with the line of your robots.txt that decides it, so you know exactly what to change, and whether you meant it.
04
It checks your llms.txt and meta description
It reads your llms.txt, if you have one, for the parts the format expects and for whether AI crawlers may read it, and your page's title and meta description, measured in pixels the way Google cuts them.
The crawlers
The AI crawlers we check, and what each one does
Three kinds of crawler read your site for AI, and they are worth treating differently. Search crawlers index pages for the answers assistants give today, and link back. On-request fetchers read a page when a person asks the assistant about it. Training crawlers collect pages to train future models. Blocking the first two keeps you out of answers; blocking the third keeps your pages out of training. And two search engines — Google and Bing — feed AI answers from their own indexes.
| User-agent | Company | What it does |
|---|---|---|
| AI search and answers | ||
OAI-SearchBot |
OpenAI | Indexes pages so ChatGPT search can show and link them. Not used for training. |
Claude-SearchBot |
Anthropic | Indexes pages to improve the search results in Claude's answers. |
PerplexityBot |
Perplexity | Indexes pages for Perplexity's answers, which cite and link their sources. |
DuckAssistBot |
DuckDuckGo | Fetches pages to write DuckDuckGo's AI-assisted answers. |
| Fetching a page a person asks about | ||
ChatGPT-User |
OpenAI | Fetches a page when a ChatGPT user asks about it or shares a link. |
Claude-User |
Anthropic | Fetches a page when a Claude user asks about it. |
Perplexity-User |
Perplexity | Fetches a page when a Perplexity user asks about it. |
MistralAI-User |
Mistral AI | Fetches a page when a user of Mistral's Le Chat asks about it. |
| AI training | ||
GPTBot |
OpenAI | Collects pages to train OpenAI's models. |
ClaudeBot |
Anthropic | Collects pages to train Anthropic's Claude models. |
Google-Extended |
Not a crawler but a token: decides whether pages Google crawls may be used for its Gemini models. Does not affect Google Search. | |
Applebot-Extended |
Apple | A token: decides whether pages Applebot crawls may train Apple's AI models. Does not affect Siri or Spotlight. |
Meta-ExternalAgent |
Meta | Collects pages for Meta's AI, including training its models. |
Amazonbot |
Amazon | Amazon's crawler, used to improve its services, including training AI models. |
CCBot |
Common Crawl | Common Crawl's crawler. Its free web archive is a common source of AI training data. |
Bytespider |
ByteDance | ByteDance's crawler, used to train its AI models. |
| Search engines AI answers draw on | ||
Googlebot |
Google Search, including AI Overviews and AI Mode. | |
Bingbot |
Microsoft | Bing search, which Microsoft Copilot's answers draw on. |
Allow or block
How to allow or block AI crawlers in robots.txt
Robots.txt is a plain text file at the root of your site. Each group starts with the crawlers it is for (User-agent) and lists what they may not read (Disallow) or may (Allow). Three common set-ups:
Let every crawler in
The default, and the same as having no robots.txt at all.
User-agent: *
Allow: /Keep AI training out, stay in AI answers
Turns away the crawlers that collect pages for training, and lets in the ones that put you in answers and send visitors.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /Turn away one crawler
One group names it; everything else is unchanged.
User-agent: GPTBot
Disallow: /Free · No account · A few seconds
Also checked
Your llms.txt and your meta description
The two other things an assistant or a search engine reads about your site before anything else.
llms.txt
Whether your site has one, whether it has the parts the format expects (a # name, a > summary and lists of links), and whether your robots.txt lets the AI crawlers read it. An llm.txt, a letter short, is flagged too.
Generate an llms.txt for your siteMeta description
Your page's title and meta description as Google may show them, on a computer and on a phone. Google cuts them by width, not by count, so they are measured in pixels. Rewrite them in the result and watch the preview change.
| What | Width | About |
|---|---|---|
| Description, on a computer | 920 px | 155 characters |
| Description, on a phone | 680 px | 120 characters |
| Title | 600 px | 55 to 60 characters |
Widths are estimated from Arial's letter widths, as most checkers do, and Google rewrites many descriptions to fit the search.
FAQ
Questions about AI crawlers and robots.txt
How do I check whether my robots.txt blocks AI crawlers?
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
Should I block AI crawlers?
Does blocking Google-Extended remove my site from Google Search or AI Overviews?
Does robots.txt stop AI companies from using my content?
What happens if my robots.txt returns an error?
What is llms.txt?
Is the crawler check free?
How long should a meta description be?
Why does the crawler check look at my meta description?
Can ChatGPT see my website?
Next
Crawlers can read you. Does ChatGPT name you?
Being readable is the first step. The free ChatGPT visibility check asks ChatGPT what your customers ask and shows whether it names you — and who it names instead.