AI Crawler Checker: Make sure AI can read your site.

An AI crawler checker shows whether ChatGPT, Perplexity, Google AI Overviews and other AI models can read your site. Referenced reads your robots.txt against all 29 AI crawlers every day, checks llms.txt and your homepage for noindex, and turns anything blocking AI into a fix.

Start your free trial

Free 7-day trial · no card · plans from $79/mo

Model Access

Checked today

4 of 5 models can read your site

ChatGPT 18 pages

AI Overviews 21 pages

Perplexity Blocked

Gemini Not seen yet

Claude 6 pages

3 models cited you in the last 30 days

Illustrative example. Tarnwell and its competitors are invented brands, and every figure is sample data.

Every AI crawler checked

Your robots.txt is read against 29 AI crawlers, with a verdict for each and the rule that decided it.

A status for every AI model

Each model you track shows as Cited, Not seen yet, Blocked or Unknown, with the pages it cited in the last 30 days.

Files and homepage tags

See whether you have a robots.txt and an llms.txt, and whether your homepage carries noindex or nosnippet.

A fix for anything blocking AI

A blocked crawler, a partial block, a homepage noindex or a robots.txt that will not load becomes a recommendation.

Model access

Can ChatGPT, Perplexity and Google AI read your site?

It depends on your robots.txt and your homepage tags. Referenced gives every AI model you track one of four statuses, so you can see which can read your site and which have cited it.

Blocked wins over everything else. A model is Blocked when a crawler it relies on is disallowed at the root of your robots.txt, or when your homepage has a noindex tag and the model uses Google's index (Google AI Overviews and AI Mode). ChatGPT counts as Blocked by a noindex tag only while OAI-SearchBot can still crawl the page to read it. A block on only some pages never makes a model Blocked; it shows on the crawler's row instead.

Microsoft Copilot has no crawler of its own, because it takes its web answers from Bing. It is not scored as a model: the Bingbot row in the crawler list carries a Copilot note, including any homepage tag that keeps Copilot out.

Cited

The model cited one of your pages in the last 30 days.
Not seen yet

Nothing blocks it, but it has not cited a page of yours. That usually means it chose other sources.
Blocked

Your robots.txt shuts out a crawler it relies on, or a homepage noindex keeps you out of its index. A blocked model can still have cited you recently, because a block stops new reads, not past citations.
Unknown

Your robots.txt could not be read, so we cannot say either way.

Crawler permissions

Which AI crawlers should you allow in robots.txt?

Allow the crawlers that let AI answer with your pages, and block the ones that only collect training data if you want to. Referenced shows each crawler's purpose beside your rule, so the choice is yours.

The list covers 29 crawlers, sorted into Answers, Search, Training, User fetch, Assistant, Ads and Dataset. Answer and search crawlers, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, decide whether AI can use your pages in a reply. Blocking a training crawler such as GPTBot or ClaudeBot is a separate choice about training, so it shows in neutral grey, not red.

Each row shows the rule that decided it: named in robots.txt, covered by the * rule, or no rule, so open by default. A crawler that is allowed but blocked from some pages shows how many, from your sitemap (up to 400 pages, and the table says so when it is cut short). User fetch crawlers act when a person asks AI to open a page, and some, such as ChatGPT-User and Perplexity-User, may ignore robots.txt; the table says so.

Referenced reads your robots.txt and never edits it. A blocked row links to its fix, which says what to change in your own file.

Crawler Permissions

All types

Answers 5 Search 6 Training 8 User fetch 6 Assistant 1 Ads 1 Dataset 2

Crawler	Type	Rule	Access	Actions
GPTBot OpenAI's crawler for training its models. Training · Named in robots.txt	Training	Named in robots.txt	Blocked
OAI-SearchBot OpenAI's crawler for ChatGPT's search results. Answers · Named in robots.txt Blocked on 14 of 212 pages e.g. /outlet/, /outlet/storm-trail-shoe, /outlet/gaiters	Answers	Named in robots.txt	Allowed	See fix: Let OAI-SearchBot read your outlet pages
ChatGPT-User Fetches a page when a ChatGPT user asks for it. May ignore robots.txt. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
OAI-AdsBot Checks the landing pages of ChatGPT ads. Ads · Covered by the * rule	Ads	Covered by the * rule	Allowed
ClaudeBot Anthropic's crawler for training its models. Training · Named in robots.txt	Training	Named in robots.txt	Blocked
Claude-SearchBot Anthropic's crawler for Claude's search results. Answers · Covered by the * rule	Answers	Covered by the * rule	Allowed
Claude-User Fetches a page when a Claude user asks for it. Respects robots.txt. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
PerplexityBot Perplexity's crawler for its answers and search results. Not used for training. Answers · Named in robots.txt	Answers	Named in robots.txt	Blocked	See fix: Allow PerplexityBot to crawl your site
Perplexity-User Fetches a page when a Perplexity user asks for it. May ignore robots.txt. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
Google-Extended Not a crawler: a robots.txt token deciding whether Google may use your pages for Gemini training and grounding. Training · Covered by the * rule	Training	Covered by the * rule	Allowed
Googlebot Google Search's crawler. Also feeds AI Overviews and AI Mode. Search · Covered by the * rule	Search	Covered by the * rule	Allowed
Applebot Apple's crawler for Siri, Spotlight and Safari search. Its pages may also train Apple's AI unless Applebot-Extended is blocked. Search · Covered by the * rule	Search	Covered by the * rule	Allowed
Applebot-Extended Not a crawler: a robots.txt token deciding whether Apple may train its AI models on your pages. Training · Covered by the * rule	Training	Covered by the * rule	Allowed
Bingbot Microsoft's Bing crawler. Bing also powers Copilot's web answers; to keep Copilot out but stay in Bing search, use noarchive. Search · Covered by the * rule No homepage tag limits Copilot.	Search	Covered by the * rule	Allowed
Bravebot Brave Search's crawler. Brave says it uses no distinct user agent, so a Bravebot rule may not apply. Search · Covered by the * rule	Search	Covered by the * rule	Allowed
Amazonbot Amazon's crawler, used to improve its products and possibly to train its AI models. Assistant · Covered by the * rule	Assistant	Covered by the * rule	Allowed
Amzn-SearchBot Amazon's crawler for search in Alexa and Rufus. Not used to train its AI models. Search · Covered by the * rule	Search	Covered by the * rule	Allowed
Amzn-User Fetches a page when an Amazon assistant user asks for it. May ignore robots.txt. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
Meta-ExternalAgent Meta's crawler for training its AI models. Training · Covered by the * rule	Training	Covered by the * rule	Allowed
Meta-WebIndexer Meta's crawler for Meta AI's search results. Search · Covered by the * rule	Search	Covered by the * rule	Allowed
Meta-ExternalFetcher Fetches a page when a Meta AI user asks for it. May ignore robots.txt. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
Bytespider ByteDance's crawler, reportedly for training its AI models. Training · Covered by the * rule	Training	Covered by the * rule	Allowed
Cohere-AI Crawler run by Cohere, an AI model company. Cohere does not document it. Training · Covered by the * rule	Training	Covered by the * rule	Allowed
D Diffbot Builds a knowledge graph of the web from what it crawls. Dataset · Covered by the * rule	Dataset	Covered by the * rule	Allowed
C CCBot Common Crawl's crawler. Its free web archive is widely used to train AI models. Dataset · Named in robots.txt	Dataset	Named in robots.txt	Blocked
D DuckAssistBot DuckDuckGo's crawler for its AI-assisted answers. Not used for training. Answers · Covered by the * rule	Answers	Covered by the * rule	Allowed
M MistralAI-User Fetches a page when a Le Chat user asks for it. Not used for training. User fetch · Covered by the * rule	User fetch	Covered by the * rule	Allowed
M MistralAI-Index Mistral's crawler for Le Chat's search results. Not used for training. Answers · Covered by the * rule	Answers	Covered by the * rule	Allowed
M MistralAI-Training Mistral's crawler for training its models. Training · Covered by the * rule	Training	Covered by the * rule	Allowed

212 sitemap pages checked. Blind spots: firewall and CDN rules (such as Cloudflare's AI bot blocking) aren't visible to us.

Files and directives

Does llms.txt matter?

Not yet: no AI model documents using it. llms.txt is a proposed convention, a plain-text guide to your site written for AI, and Google says AI Overviews and AI Mode need no AI text files. Referenced checks for one and shows the result, but a missing llms.txt is neutral grey, never a problem, and no recommendation asks you to add it.

The same card shows your robots.txt, with a View button that opens the live robots.txt or llms.txt from your site, and the tags on your homepage that can keep AI away: noindex, which takes the page out of Google's AI features and out of ChatGPT while OAI-SearchBot can crawl it, and nosnippet, which limits the text shown. Noarchive, nocache and a Bing-only noindex limit Microsoft Copilot and appear on the Bingbot row. Only the homepage is checked for these.

If your robots.txt has Content-Signal lines, they show too. They are requests to AI companies that nobody can verify are honoured, so Referenced flags where your own rules undercut them, such as ai-train=no while training crawlers are still allowed. Blocking a crawler is what keeps it out.

Files and Directives

Crawler Rules /robots.txt

Tells crawlers which paths they may read.

Content signals not enforced

Found

AI Site Guide /llms.txt

A short guide for AI agents to what your site covers. Proposed convention. Google says AI Overviews and AI Mode do not need it.

Not found

Homepage Directives

Meta robots and X-Robots-Tag on your homepage.

None found

robots.txt

Live

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /outlet/

User-agent: *
Disallow: /cart/
Content-Signal: ai-train=no

Content signals are as of the last check. Training crawlers such as Google-Extended are still allowed, so the ai-train=no request is not enforced.

How checks run

How often does Referenced check your AI access?

Every day, around the time your scan runs, on every plan. Your first check runs with your first scan, so the page has data from the start.

Press Re-check after you change something, up to 3 times an hour. It refreshes your robots.txt, llms.txt and homepage but does not re-read your sitemap; the scheduled check does that now and then.

A check that fails because of our own network trouble is never counted against your site. While a check is under way the page header shows Checking, and Delayed or Stalled if it is late or stuck. If a crawler that could read your site becomes blocked, you get a notification in the app. Each site keeps 90 days of check history.

How Checks Run

AI Access

Schedule Every day, around your scan time

First check Runs with your first scan

Re-check Up to 3 an hour

History 90 days of checks

Our network trouble Never counted against you

A crawler becomes blocked Notification in the app

Header pill while a check is Checking Delayed Stalled

Read from your live robots.txt, llms.txt and homepage.

Recommendations

How do you fix a blocked AI crawler?

Open the See fix link on the blocked row, or find it in Recommendations, where it is marked AI access. It says what to change in your robots.txt or on your homepage.

Four things raise an AI access recommendation: a blocked answer crawler, an answer crawler blocked from part of your sitemap, a noindex tag on your homepage, and a robots.txt that keeps failing to load. They are ranked with the rest of your recommendations by priority.

You make the change in your own site, then press Re-check to see the new verdict.

Open Recommendations

High priority Allow PerplexityBot to crawl your site Your robots.txt blocks the crawler Perplexity uses, so it cannot read your pages or cite them. AI access
Medium priority Let OAI-SearchBot read your outlet pages The outlet pages in your sitemap are blocked for OAI-SearchBot, so ChatGPT search cannot read them. AI access

AI crawler checker questions, answered.

ChatGPT, GPTBot, robots.txt, llms.txt and how often Referenced checks.

Can ChatGPT crawl my site? +

Yes, unless your robots.txt blocks OAI-SearchBot, the crawler behind ChatGPT search, or your homepage carries a noindex tag. Referenced checks both and shows ChatGPT as Cited, Not seen yet or Blocked. A firewall or CDN can also stop crawlers when your robots.txt allows them, and Referenced cannot see those rules.

What is the difference between GPTBot and OAI-SearchBot? +

GPTBot collects content to train OpenAI's models. OAI-SearchBot finds pages for ChatGPT's search results. Blocking GPTBot keeps your pages out of training data but does not remove you from ChatGPT search; blocking OAI-SearchBot is what can. A third crawler, ChatGPT-User, fetches a page when a person asks for it and may ignore robots.txt. Referenced shows each one's purpose beside your rule.

Should I block AI training crawlers? +

That is your call. Blocking training crawlers such as GPTBot or ClaudeBot stops them using your pages to train models, and it does not affect whether AI can cite you, because answers come from the search crawlers. Referenced shows a blocked training crawler in neutral grey, not red. Keep the answer and search crawlers allowed, and check Google-Extended before blocking it: it is the token Google uses for Gemini training and grounding.

Does llms.txt help? +

Not that any AI model has documented. llms.txt is a young convention, not a standard, and Google says AI Overviews and AI Mode need no AI text files. Referenced shows whether you have one as information and does not recommend adding it.

Can robots.txt stop ChatGPT citing me? +

Yes, if it blocks OAI-SearchBot, the crawler ChatGPT uses to find pages. A noindex tag on your homepage can too: OpenAI says a noindex keeps a page out of ChatGPT search, and OAI-SearchBot must be able to crawl the page to read the tag, so Referenced shows ChatGPT as Blocked in that case. A CDN or firewall can also block crawlers even when your robots.txt allows them, so a clean robots.txt is a good start, not a guarantee.

How do I check if my site blocks AI crawlers? +

Open AI Access in Referenced. The Crawler Permissions list shows Allowed or Blocked for each of 29 AI crawlers, the rule that decided it and, for an allowed crawler, how many sitemap pages it is blocked from. Referenced applies the usual robots.txt rules: a group naming the crawler beats the * group, and the longest matching rule wins. It reads your robots.txt and never edits it.

How often does Referenced check? +

Every day, automatically, around the time your scan runs. Press Re-check whenever you change something, up to 3 times an hour. Each site keeps 90 days of check history.

Is AI Access on every plan? +

Yes. The check runs on every plan, with no extra setup, and your first check runs with your first scan.

Find out whether AI can read your site.

Free 7-day trial, no card: your first AI Access check runs with your first scan.

Feature

Visibility

Your score across every AI model.

Feature

Sentiment

What AI thinks of your brand.

Feature

Competitors

Share of voice, brand by brand.

Feature

Sources

Which pages AI actually cites.

Feature

AI Access

Can AI crawlers read your site?

Feature

Recommendations

The fix for every gap, ranked.