robots.txt for AI crawlers: which bots to allow
robots.txt is a plain text file at yourdomain.com/robots.txt. It tells software which parts of your site it may read. Most small-business sites have a single general rule and nothing else. That leaves the AI companies to guess, and each of them runs more than one crawler.
The important split is between two jobs.
- Search bots read your pages so an AI assistant can find and cite you in an answer.
- Training bots collect pages to train future models.
You can allow one job and block the other.
The crawlers
| Crawler | Company | Job |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search |
| GPTBot | OpenAI | Collects pages for model training |
| ChatGPT-User | OpenAI | Fetches a page when a person asks |
| Claude-SearchBot | Anthropic | Finds pages for Claude’s search |
| ClaudeBot | Anthropic | Collects pages for model training |
| Claude-User | Anthropic | Fetches a page when a person asks |
| PerplexityBot | Perplexity | Finds pages for Perplexity search |
| Perplexity-User | Perplexity | Fetches a page when a person asks |
| Google-Extended | Opts out of Gemini training; does not affect Google Search | |
| Applebot-Extended | Apple | Opts out of Apple Intelligence training |
A file that allows search and blocks training
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xml
Want to allow training as well? Change the four Disallow: / lines to Allow: /. The choice is yours. Blocking training bots does not remove you from search answers.
Three things that catch people out
The user bots may ignore the file. ChatGPT-User and Perplexity-User act when a person asks a question. Both companies say this kind of request may not follow robots.txt.
A firewall can override you. Your host or a service in front of your site may block unknown bots by default. Then robots.txt says welcome and the firewall shuts the door. Ask your host to confirm in writing that OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed through.
The file is a request, not a lock. Well-behaved bots follow it. Nothing forces every bot to. Do not use robots.txt to hide anything private.
Check your file
- Open
yourdomain.com/robots.txtin a browser. It should load, not show “not found”. - Look for a line that says only
Disallow: /underUser-agent: *. That blocks everything. Remove it. - Confirm the last line points at your sitemap.
Crawler names change. Recheck the list every six months.
This is one of 52 checks in the Business Website Launch Checklist, in the chapter Get Mentioned by AI Tools.