Free robots.txt for AI crawlers: Build a robots.txt for AI crawlers
Choose which AI crawlers may read your site, grouped by what they do. Copy or download the lines to add to your robots.txt. It runs in your browser and sends nothing anywhere.
Choose Your Rules
Pick one rule for every AI crawler.
Options
Content-Signal line
Adds your search, AI answer and AI training preferences in a User-agent: * group.
Category comments
Puts a short comment above each group of crawlers.
Date comment
Adds a line at the top saying when the file was generated.
Adds a Sitemap line so crawlers find your pages. Use the full address.
Your robots.txt
Add these lines to the robots.txt you already have.
Paste these lines into your current robots.txt and keep everything already there.
A crawler listed here follows only its own lines and skips your User-agent: * rules. If those rules keep crawlers out of private folders, copy the Disallow lines into its group too.
Once the file is live, test it with our free AI crawler checker. Check your site
How to Publish Your robots.txt and Check It
Five steps from the lines above to a live file that every crawler can find and read, then a check that each one gets the answer you meant.
Add the lines
Paste them into your existing robots.txt. Keep its other rules, but delete any old group for these crawlers.
Save as plain text
Name the file robots.txt and save it as plain UTF-8 text, not a word processor document or web page.
Upload it to the root
It must load at yourdomain.com/robots.txt. Each subdomain, such as blog.yourdomain.com, needs its own copy.
Open it in a browser
You should see plain text, not a web page. A server error here can make crawlers treat the site as blocked.
Run the free checker
Crawlers can take up to a day to fetch it. Then run the AI crawler checker to see what each one can reach.
Check your CDN too. If Cloudflare's AI Crawl Control blocks a crawler, it is turned away before it reaches your robots.txt, so these lines alone will not let it in.
Where robots.txt Goes on Your Platform
Each platform has one right place for robots.txt, and most have a catch. Find yours below, add the lines there, then load yourdomain.com/robots.txt to confirm they are live.
| Platform | Where It Goes | Watch Out For |
|---|---|---|
| WordPress | Your SEO plugin's robots.txt editor, or a robots.txt file uploaded to the site root. | An uploaded file replaces the virtual one WordPress generates, so edit one or the other, not both. |
| Shopify | A robots.txt.liquid template added to your theme. | Shopify ships a sensible default. Add your lines to it rather than replacing it, so Shopify can keep it current. |
| Static sites and Cloudflare Pages | The public or build output folder, so it deploys to the site root. | Check the live URL after you deploy, not just the file in your project. |
| Nginx and Apache | The document root of each site or virtual host. | Each subdomain is a separate host and needs its own file. |
| Wix | The Robots.txt Editor in Wix's SEO Tools. | Wix warns that edits can drop pages from search and that it won't help fix them, so change only what you need. |
What Each Option Adds to Your File
Your Allow and Block choices write the rules. The options add a few lines around them, and only the Disallow lines ever keep a crawler out.
Content-Signal Line
Cloudflare introduced Content-Signal in September 2025. It states how you are happy for your pages to be used once a crawler has fetched them: for search, for AI answers (ai-input) and for AI training (ai-train), each yes or no.
It is a preference, not a rule. It is not part of RFC 9309, crawlers may ignore it, and in July 2026 Google's John Mueller said he knew of no crawler or LLM that uses it. It sits in its own User-agent: * group, so it never replaces your crawler groups.
User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap URL
Points crawlers to your sitemap. The line is not tied to a crawler group, so one line serves every crawler that reads it. Use the full URL, one line per sitemap.
Sitemap: https://example.com/sitemap.xml
Category Comments
Labels each group so the next person to edit the file can see what it is for. Crawlers ignore everything after a # on a line.
# Training and datasets User-agent: GPTBot Disallow: /
Date Comment
Records when you generated the lines, so you can tell which version is live. It has no effect on crawlers.
# Generated by Referenced AI on 2026-11-06Questions About robots.txt and AI Crawlers
Does blocking GPTBot stop ChatGPT search? + −
No. GPTBot collects pages to train OpenAI's models. ChatGPT search uses OAI-SearchBot, so you can block GPTBot and stay in ChatGPT search results.
What does Google-Extended control? + −
It decides whether Google may use your pages for Gemini training and grounding. It does not affect Google Search, and Googlebot still crawls you either way. Blocking Googlebot is a different choice: it also removes you from Google Search.
Do all AI agents obey robots.txt? + −
No. Crawlers that collect pages in bulk generally do. Agents that fetch a single page because a person asked, such as ChatGPT-User and Perplexity-User, may ignore it. Those rows are marked in the builder.
Should I block training crawlers? + −
It is your call. Blocking them asks AI companies not to use your pages to train new models, and it costs you nothing in search. It does not remove what a model already learned, and a model can still describe your brand from other sources.
How do I use the file? + −
Add the lines to the robots.txt at the root of your domain, so it is served at yourdomain.com/robots.txt. If you already have one, paste them in beside your existing rules. Then check the result with the free AI crawler checker.
Want to know if it worked?
Free 7-day trial, no card: Referenced checks your crawler access every day and tracks your AI visibility.