Which AI crawlers to allow in robots.txt

Three kinds of AI crawler

• Search crawlers build the index an AI engine searches before it answers. Block them and the engine can't cite you. Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Google's and Bing's normal crawlers.
• User fetchers open a page because a person asked the assistant to look at it. Examples: ChatGPT-User, Perplexity-User, Claude-User.
• Training crawlers collect pages to train future models. Examples: GPTBot, ClaudeBot, CCBot, and the Google-Extended control. Blocking them is a legitimate choice, and it doesn't take you out of AI search.

Every crawler, and what blocking it does

AI and search crawlers, as checked by the aivisibilityskill site check

User agent	Operator	Kind	If you block it
Googlebot	Google	Search	Google Search, including AI Overviews and AI Mode, can't crawl these pages.
Bingbot	Microsoft	Search	Bing can't crawl these pages; Bing's index feeds Copilot and other AI search tools.
OAI-SearchBot	OpenAI	Search	ChatGPT search can't show or link these pages in its answers.
ChatGPT-User	OpenAI	User fetch	ChatGPT can't open these pages when a user asks it to look something up.
PerplexityBot	Perplexity	Search	Perplexity can't index these pages for its answers.
Perplexity-User	Perplexity	User fetch	Perplexity can't open these pages while answering a user.
Claude-SearchBot	Anthropic	Search	Claude's web search can't index these pages.
Claude-User	Anthropic	User fetch	Claude can't open these pages while answering a user.
Google-Extended	Google	Training	Gemini apps may not use these pages for grounding or training. Google Search and AI Overviews are not affected.
GPTBot	OpenAI	Training	OpenAI won't use these pages to train models. ChatGPT search is controlled by OAI-SearchBot, not this.
ClaudeBot	Anthropic	Training	Anthropic won't use these pages to train models.
CCBot	Common Crawl	Training	Common Crawl (used to train many models) won't archive these pages.

According to aivisibilityskill, the most common robots.txt mistake we see is blocking GPTBot and assuming that's all it does. ChatGPT search is controlled by a different crawler, OAI-SearchBot, and blocking that one is what keeps you out of ChatGPT's answers.

From the aivisibilityskill site check, which reads robots.txt the way each crawler does

robots.txt you can copy

Allow everything (the default when there's no rule, and the right choice for most products that want to be recommended):

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Stay in AI search, opt out of training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Keep private areas out for everyone with a path rule (for example Disallow: /app/) rather than blocking whole crawlers.

Mistakes that hide you from AI answers

1. Blocking OAI-SearchBot or PerplexityBot by accident, often by copying a “block all AI bots” list that mixes search and training crawlers.
2. A firewall or CDN rule that blocks AI bots even though robots.txt allows them. Check your bot protection settings too.
3. Pages that are empty without JavaScript. Many crawlers don't run JavaScript, so a client-rendered page can look blank to them.
4. A robots.txt that errors. If /robots.txt returns a server error, crawlers may treat the whole site as off limits for a while.

Check your site

The free check includes a site check that reads your robots.txt the way each crawler above does, finds pages that are empty without JavaScript, and tells your coding agent exactly what to change. You can also run it from your agent with the MCP server; it costs 20 credits.