Free AI crawler checker: Can AI crawlers read your site?
Enter a domain to see which AI crawlers its robots.txt lets in and which it blocks. It answers "Can ChatGPT crawl my site?" in a few seconds, free and with no account.
Free, no sign-up. Reads your robots.txt, llms.txt and homepage.
-
GPTBot -
ClaudeBot -
PerplexityBot -
Googlebot -
Meta-ExternalAgent -
Bingbot - +28 more crawlers, 34 in all: see the full list
How the AI crawler checker works
A check takes a few seconds. It only reads public files, so nothing on your site changes.
Enter your domain
Type it into the box above. The checker fetches a few public files, as a crawler would, and stores nothing about you.
We match every crawler to your rules
Each crawler obeys the group that names it, or the * group if none does, and the longest matching rule wins (RFC 9309). It is an easy order to get wrong by eye.
A verdict per AI model, and the fix
See which AI models can read the site and which crawlers are blocked. If an answer crawler is shut out, a link opens the robots.txt builder with the fix filled in. Every result has a page you can share.
5 of 6 AI models can read acme.co.uk
-
ChatGPT
Allowed
-
Perplexity
Blocked
-
Claude
Allowed
What invisibility costs you
When an AI crawler is shut out, the answers carry on without you.
Nobody tells you
Search Console has no report for AI crawlers, and your rankings do not move. Often the first sign is an AI answer that should have named you and did not.
Someone else gets the mention
AI tools answer from what their crawlers could fetch. Shut yours out and the answer still gets written, from a rival's pages.
One rule can cost several tools
Googlebot serves AI Overviews, AI Mode and Google Search, and Bingbot feeds Microsoft Copilot. A rule aimed at one tool can take out the others.
Illustration: the same question answered twice, once when the site blocks the AI crawler and once when it allows it.
Blocked
acme.co.uk blocks the crawler
ChatGPT
Allowed
acme.co.uk lets the crawler in
ChatGPT
Why AI crawlers get blocked by mistake
Most blocks were never a decision. They arrive with something else, like a template or a migration, and stay because nothing flags them.
Staging rules that went live
Test sites are usually hidden from every crawler. Copy that robots.txt to the live site and the block goes live with it.
User-agent: * Disallow: /
A copied block list
Lists shared to keep scrapers out often include answer crawlers too, so PerplexityBot or OAI-SearchBot is blocked along with them.
User-agent: CCBot User-agent: PerplexityBot Disallow: /
Blocking Googlebot to stop AI Overviews
Googlebot also crawls for Google Search, so this rule takes the site out of both. No robots.txt token removes AI Overviews on its own.
User-agent: Googlebot Disallow: /
A "hide from search engines" setting
Settings such as WordPress's "Discourage search engines" add noindex to every page, and engines that honour it leave the site out.
<meta name="robots" content="noindex">
A snippet limit
nosnippet stops Google quoting the page, which also keeps it out of AI Overviews and AI Mode. The page can still rank.
<meta name="robots" content="nosnippet">
Bot protection on your CDN
A CDN or security plugin can turn a crawler away whatever robots.txt says. The checker only sees whether its own request is refused.
403 Forbidden
Which AI crawlers does it check?
The checker covers 34 AI crawlers and robots.txt tokens from 17 companies, among them GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended. It is the same list the Referenced app checks every day. Each one affects whether an AI tool can read, cite or train on your pages, and each meets the four rules below.
Documented by its operator
Added only once its operator publishes the name. Two older entries predate this rule.
No third-party guesses
Names found only on third-party lists stay off.
No retired names
anthropic-ai and Claude-Web no longer crawl, so a rule for them does nothing.
No double counting
ChatGPT-User/2.0 matches the ChatGPT-User rule, so it counts once.
Answers and search
They read your pages so an AI tool or search engine can find them and cite them. Block one and that tool loses your pages as a source.
D
M
Fetch a page when a person asks
They visit a page only when someone asks for it, for example by pasting your link into ChatGPT. Some may ignore robots.txt, so treat their result as a guide.
M
Training and datasets
They collect pages to train AI models or build public datasets. Blocking them does not take you out of search or live answers.
D
C
A
W
M
Ads and other
Ad checks, shopping assistants and AI agents a site owner sets up. They have little to do with whether AI answers cite you.
Every crawler on the list 34 crawlers · reviewed + −
Answers and search
12 crawlers-
OAI-SearchBot
OpenAI's crawler for ChatGPT's search results.
-
Claude-SearchBot
Anthropic's crawler for Claude's search results.
-
PerplexityBot
Perplexity's crawler for its answers and search results. Not used for training.
-
Googlebot
Google Search's crawler. Also feeds AI Overviews and AI Mode.
-
Applebot
Apple's crawler for Siri, Spotlight and Safari search. Its pages may also train Apple's AI unless Applebot-Extended is blocked.
-
Bingbot
Microsoft's Bing crawler. Bing also powers Copilot's web answers; to keep Copilot out but stay in Bing search, use noarchive.
-
Bravebot
Brave Search's crawler. Brave says it uses no distinct user agent, so a Bravebot rule may not apply.
-
Amzn-SearchBot
Amazon's crawler for search in Alexa and Rufus. Not used to train its AI models.
-
Meta-WebIndexer
Meta's crawler for Meta AI's search results.
-
D
DuckAssistBot
DuckDuckGo's crawler for its AI-assisted answers. Not used for training.
-
D
DuckDuckBot
DuckDuckGo's search crawler.
-
M
MistralAI-Index
Mistral's crawler for Le Chat's search results. Not used for training.
Fetch a page when a person asks
6 crawlers-
ChatGPT-User
Fetches a page when a ChatGPT user asks for it. May ignore robots.txt.
-
Claude-User
Fetches a page when a Claude user asks for it. Respects robots.txt.
-
Perplexity-User
Fetches a page when a Perplexity user asks for it. May ignore robots.txt.
-
Amzn-User
Fetches a page when an Amazon assistant user asks for it. May ignore robots.txt.
-
Meta-ExternalFetcher
Fetches a page when a Meta AI user asks for it. May ignore robots.txt.
-
M
MistralAI-User
Fetches a page when a Le Chat user asks for it. Not used for training.
Training and datasets
13 crawlers-
GPTBot
OpenAI's crawler for training its models.
-
ClaudeBot
Anthropic's crawler for training its models.
-
Google-Extended
Not a crawler: a robots.txt token deciding whether Google may use your pages for Gemini training and grounding.
-
GoogleOther
Google's general crawler for its product teams' research and development fetches. Separate from Google Search.
-
Applebot-Extended
Not a crawler: a robots.txt token deciding whether Apple may train its AI models on your pages.
-
Meta-ExternalAgent
Meta's crawler for training its AI models.
-
Bytespider
ByteDance's crawler, reportedly for training its AI models.
-
Cohere-AI
Crawler run by Cohere, an AI model company. Cohere does not document it.
-
D
Diffbot
Builds a knowledge graph of the web from what it crawls.
-
C
CCBot
Common Crawl's crawler. Its free web archive is widely used to train AI models.
-
A
AI2Bot
The Allen Institute for AI's crawler for training its open models and datasets.
-
W
Webzio-Extended
Not a crawler: a robots.txt token deciding whether Webz.io may use your pages in AI training datasets.
-
M
MistralAI-Training
Mistral's crawler for training its models.
Ads and other
3 crawlers-
OAI-AdsBot
Checks the landing pages of ChatGPT ads.
-
Google-CloudVertexBot
Crawls a site when its owner builds a Vertex AI agent on it. Not used for Google Search.
-
Amazonbot
Amazon's crawler, used to improve its products and possibly to train its AI models.
Changes to the crawler list · Crawler list last reviewed
- : Added GoogleOther, Google-CloudVertexBot, DuckDuckBot, AI2Bot and Webzio-Extended, each documented by its operator.
- : Checker launched with the same crawler list the Referenced app uses.
Questions About AI Crawler Access
What does the AI crawler checker look at? + −
It fetches your robots.txt, llms.txt and homepage over HTTPS, so the site must use HTTPS. From robots.txt it works out whether each of the 34 crawlers may fetch the homepage path, /. For llms.txt it only checks that the file exists. It reads the homepage only when your robots.txt allows our crawler, looking for noindex and nosnippet in the meta robots tag or the X-Robots-Tag header. It runs no JavaScript and looks at no other page, so a crawler allowed at / can still be blocked elsewhere.
How do I know if ChatGPT can read my website? + −
Check that your robots.txt does not block OAI-SearchBot, the crawler behind ChatGPT search, and that the page has no noindex. The checker reads both (noindex on the homepage only). A firewall, CDN or security plugin can still turn the crawler away when robots.txt allows it, and the checker cannot see that. ChatGPT-User, which fetches a page when a person asks, may ignore robots.txt altogether.
How is the AI access score worked out? + −
The score is out of 100. Answer and search crawlers make up 70 points, weighted by reach: Googlebot 20, OAI-SearchBot 15, Bingbot 10, PerplexityBot 8 and Claude-SearchBot 6, with the rest sharing what is left. Person-triggered fetchers add 10, a homepage without noindex 12 and one without nosnippet 8. Training crawlers and llms.txt are not scored, and a site whose robots.txt cannot be read gets no score.
Does the checker store my URL or need an account? + −
It is free and needs no account. It keeps one result per domain for 90 days so the result page can be shared, and stores nothing about the person who ran the check. If you own the domain and want its result removed, use the link on the result page.
Why would blocking AI crawlers hurt my visibility? + −
A blocked answer crawler, such as OAI-SearchBot, PerplexityBot or Claude-SearchBot, cannot fetch your pages to cite them. A model can still mention you from other sources. Google AI Overviews rely on Googlebot, and blocking it also removes you from Google Search.
Should I allow or block AI crawlers? + −
Allow the answer and search crawlers if you want to be cited in AI answers. Blocking training and dataset crawlers such as GPTBot or CCBot does not remove you from search or live answers, but it may reduce what future models learn about you. The robots.txt builder writes the file for whichever mix you choose.
How often should I check? + −
Check again after any change to your hosting, CDN or firewall, security plugins, CMS or robots.txt, since any of them can flip a crawler from allowed to blocked. A domain checked in the last day shows its previous result rather than a new one. For a daily check that warns you when an answer crawler becomes blocked, use Referenced.
What does "Couldn't check" mean? + −
It means we could not get a usable robots.txt, so no verdict is shown. The site may have refused our checker with a 401 or 403, or it may not have answered at all: a timeout, an HTTP-only site, or one with no IPv4 address. It does not mean GPTBot is blocked. If a firewall is refusing our checker, allowing ReferencedBot lets the check run.
Does an Allowed result mean the crawler has visited my site? + −
No. Allowed means your robots.txt would let that crawler in, not that it has come. Your server logs show which crawlers have actually visited, and when.
Does a blocked result mean AI will never mention my brand? + −
No. A model can still describe your brand from other sources, such as reviews, directories and news coverage. Blocking only limits what an engine can fetch from your own site.
Can I track this over time? + −
Yes. Referenced checks your robots.txt every day and warns you when an answer crawler turns from allowed to blocked. It also shows how your brand appears in AI answers, so you can see whether access changes line up with visibility.
Want this checked every day?
Free 7-day trial, no card: Referenced watches your crawler access and your AI visibility.