avarisk.com
AI tools can read this site, so its pages can be quoted and cited in their answers.
Checked 7 hours ago
AI access score
100 out of 100
AI models
ChatGPT
Can read
AI Overviews
Can read
Perplexity
Can read
Claude
Can read
Gemini
Can read
AI Mode
Can read
Crawler by Crawler
Answers and search 12 crawlers + −
| Crawler | Type | Rule | Access | Fix |
|---|---|---|---|---|
|
OAI-SearchBot
OpenAI's crawler for ChatGPT's search results. Answers · Covered by the * rule |
Answers | Covered by the * rule | Allowed | |
|
Claude-SearchBot
Anthropic's crawler for Claude's search results. Answers · Covered by the * rule |
Answers | Covered by the * rule | Allowed | |
|
PerplexityBot
Perplexity's crawler for its answers and search results. Not used for training. Answers · Covered by the * rule |
Answers | Covered by the * rule | Allowed | |
|
D
DuckAssistBot
DuckDuckGo's crawler for its AI-assisted answers. Not used for training. Answers · Covered by the * rule |
Answers | Covered by the * rule | Allowed | |
|
M
MistralAI-Index
Mistral's crawler for Le Chat's search results. Not used for training. Answers · Covered by the * rule |
Answers | Covered by the * rule | Allowed | |
|
Googlebot
Google Search's crawler. Also feeds AI Overviews and AI Mode. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed | |
Applebot
Apple's crawler for Siri, Spotlight and Safari search. Its pages may also train Apple's AI unless Applebot-Extended is blocked. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed | |
Bingbot
Microsoft's Bing crawler. Bing also powers Copilot's web answers; to keep Copilot out but stay in Bing search, use noarchive. Search · Covered by the * rule No homepage tag limits Copilot. |
Search | Covered by the * rule | Allowed | |
Bravebot
Brave Search's crawler. Brave says it uses no distinct user agent, so a Bravebot rule may not apply. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed | |
Amzn-SearchBot
Amazon's crawler for search in Alexa and Rufus. Not used to train its AI models. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed | |
Meta-WebIndexer
Meta's crawler for Meta AI's search results. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed | |
|
D
DuckDuckBot
DuckDuckGo's search crawler. Search · Covered by the * rule |
Search | Covered by the * rule | Allowed |
Fetch a page when a person asks 6 crawlers + −
| Crawler | Type | Rule | Access | Fix |
|---|---|---|---|---|
|
ChatGPT-User
Fetches a page when a ChatGPT user asks for it. May ignore robots.txt. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed | |
|
Claude-User
Fetches a page when a Claude user asks for it. Respects robots.txt. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed | |
|
Perplexity-User
Fetches a page when a Perplexity user asks for it. May ignore robots.txt. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed | |
|
Amzn-User
Fetches a page when an Amazon assistant user asks for it. May ignore robots.txt. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed | |
|
Meta-ExternalFetcher
Fetches a page when a Meta AI user asks for it. May ignore robots.txt. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed | |
|
M
MistralAI-User
Fetches a page when a Le Chat user asks for it. Not used for training. User fetch · Covered by the * rule |
User fetch | Covered by the * rule | Allowed |
Training and datasets 13 crawlers + −
| Crawler | Type | Rule | Access | Fix |
|---|---|---|---|---|
|
GPTBot
OpenAI's crawler for training its models. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
ClaudeBot
Anthropic's crawler for training its models. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
Google-Extended
Not a crawler: a robots.txt token deciding whether Google may use your pages for Gemini training and grounding. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
Applebot-Extended
Not a crawler: a robots.txt token deciding whether Apple may train its AI models on your pages. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
Meta-ExternalAgent
Meta's crawler for training its AI models. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
Bytespider
ByteDance's crawler, reportedly for training its AI models. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
Cohere-AI
Crawler run by Cohere, an AI model company. Cohere does not document it. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
A
AI2Bot
The Allen Institute for AI's crawler for training its open models and datasets. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
M
MistralAI-Training
Mistral's crawler for training its models. Training · Covered by the * rule |
Training | Covered by the * rule | Allowed | |
|
GoogleOther
Google's general crawler for its product teams' research and development fetches. Separate from Google Search. Dataset · Covered by the * rule |
Dataset | Covered by the * rule | Allowed | |
|
D
Diffbot
Builds a knowledge graph of the web from what it crawls. Dataset · Covered by the * rule |
Dataset | Covered by the * rule | Allowed | |
|
C
CCBot
Common Crawl's crawler. Its free web archive is widely used to train AI models. Dataset · Covered by the * rule |
Dataset | Covered by the * rule | Allowed | |
|
W
Webzio-Extended
Not a crawler: a robots.txt token deciding whether Webz.io may use your pages in AI training datasets. Dataset · Covered by the * rule |
Dataset | Covered by the * rule | Allowed |
Ads and other 3 crawlers + −
| Crawler | Type | Rule | Access | Fix |
|---|---|---|---|---|
|
Google-CloudVertexBot
Crawls a site when its owner builds a Vertex AI agent on it. Not used for Google Search. Assistant · Covered by the * rule |
Assistant | Covered by the * rule | Allowed | |
|
Amazonbot
Amazon's crawler, used to improve its products and possibly to train its AI models. Assistant · Covered by the * rule |
Assistant | Covered by the * rule | Allowed | |
|
OAI-AdsBot
Checks the landing pages of ChatGPT ads. Ads · Covered by the * rule |
Ads | Covered by the * rule | Allowed |
Files and Directives
Crawler rules /robots.txt
Tells crawlers which paths they may read.
AI site guide /llms.txt
A short guide for AI agents to what your site covers.
Homepage directives
Meta robots and X-Robots-Tag on your homepage.
About This Check
Checked (6 Nov 2026, 10:02 UTC) against the 34 crawlers on our list.
- Rules are read for your homepage only. Other pages can be treated differently.
- Allowed means robots.txt lets a crawler in, not that it has visited.
- Fetchers that act when a person asks may ignore robots.txt.
We keep this result for 90 days and nothing about you. Own this domain? Ask us to remove it How the checker works
Share This Result
Add a Badge to Your Site
Paste this into your site's footer or about page. The badge shows your current AI Access score, links to this result and keeps itself up to date.
Changes appear within a day of a re-check.
How visible is your brand in AI answers?
Tell us your domain and we'll send you a visibility report. Free, no card.