AI crawlers list: every AI bot, what it does and how to control it
By ElixRank Team · Updated
Short answer
AI companies use three kinds of bots: search crawlers that index pages for AI answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), training crawlers that collect data for models (GPTBot, ClaudeBot, CCBot), and user-triggered fetchers that load a page when someone asks (ChatGPT-User, Claude-User, Perplexity-User). To stay visible in AI search while opting out of training, allow the search crawlers and block the training ones in robots.txt.
The three kinds of AI bots
- Search crawlers build the index an AI assistant searches when it answers. Block them and you disappear from that product's answers and citations.
- Training crawlers collect content that may be used to train models. Blocking them doesn't remove you from AI search results.
- User-triggered fetchers load a specific page because a user asked about it or pasted the link. Vendors say robots.txt may not apply to them, because a person initiated the request.
Some names aren't crawlers at all. Google-Extended and Applebot-Extended are robots.txt control tokens: no bot visits under those names. They tell Google and Apple how content their normal crawlers already fetched may be used.
AI crawler and user-agent list
| Robots.txt token | Company | Type | Notes |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | Decides whether pages can appear in ChatGPT search answers |
| GPTBot | OpenAI | Training | Opt out of model training. Independent of OAI-SearchBot |
| ChatGPT-User | OpenAI | User fetch | User-initiated visits from ChatGPT and GPTs; OpenAI says robots.txt rules may not apply |
| Claude-SearchBot | Anthropic | Search | Indexes pages to improve Claude's search results |
| ClaudeBot | Anthropic | Training | Collects content that may be used for training |
| Claude-User | Anthropic | User fetch | Fetches pages when Claude users ask; blocking can reduce visibility in user-directed search |
| PerplexityBot | Perplexity | Search | Builds Perplexity's index; Perplexity says it isn't used for training |
| Perplexity-User | Perplexity | User fetch | Loads pages for live questions; Perplexity says it generally ignores robots.txt |
| Googlebot | Search | Powers Google Search, including AI Overviews and AI Mode | |
| Google-Extended | Control token | Opts out of Gemini training and grounding in Gemini apps and Vertex AI; doesn't affect Search or AI Overviews | |
| Bingbot | Microsoft | Search | Powers Bing, which grounds Microsoft Copilot answers |
| Applebot | Apple | Search | Siri and Spotlight suggestions |
| Applebot-Extended | Apple | Control token | Opts out of Apple's generative AI training; normal Applebot indexing continues |
| Meta-ExternalAgent | Meta | Training | Model training and indexing for Meta AI |
| Meta-ExternalFetcher | Meta | User fetch | Fetches links users share with Meta AI; Meta says it may bypass robots.txt |
| CCBot | Common Crawl | Training | Open web archive widely used to train models |
| Bytespider | ByteDance | Training | Data for ByteDance models; widely reported to ignore robots.txt |
| Amazonbot | Amazon | Search / AI | Used for Alexa and other Amazon services |
| DuckAssistBot | DuckDuckGo | User fetch | Fetches pages for DuckAssist answers |
robots.txt rules to copy
Recommended for most sites: stay in AI search, opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
# Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot,
# Googlebot, Bingbot) aren't listed, so they follow your * rules.
User-agent: *
Allow: /Block AI search too (you'll lose AI citations from these products)
User-agent: OAI-SearchBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /A crawler obeys only the most specific group that names it. If you add a group for OAI-SearchBot, it ignores the * group, so repeat any shared Disallow rules (like /admin/) inside it. Test your file against each bot with the robots.txt tester, which has AI crawler presets.
How to check which AI bots you're blocking
- Run your homepage through the AI visibility audit. It reads robots.txt and lists AI search and training bots separately.
- Check your CDN or firewall too. Cloudflare and other CDNs can block AI bots at the edge regardless of robots.txt, and some sites have this turned on without knowing.
- Look at server logs for the user-agent names above to see which bots actually visit and how often.
Background on the file itself: how robots.txt works. Why access matters for citations: how AI search chooses sources.
Should you block AI crawlers?
| If you… | Then |
|---|---|
| Want traffic and citations from ChatGPT, Claude and Perplexity | Allow their search crawlers |
| Don't want your content used to train models | Block the training crawlers and control tokens; it doesn't cost you AI search visibility |
| Sell content behind a paywall or licence | Block training crawlers; consider blocking search crawlers too, and serve only previews to bots |
| Run a small business or SaaS site | Allow search crawlers; training is your call. Visibility usually matters more than the opt-out |
Frequently asked questions
What is GPTBot?
OpenAI's training crawler. It collects content that may be used to train OpenAI's models. Blocking it in robots.txt opts you out of training without affecting ChatGPT search, which uses OAI-SearchBot.
What is OAI-SearchBot?
OpenAI's search crawler. It decides whether your pages can appear and be cited in ChatGPT search answers. Block it only if you don't want to appear in ChatGPT search.
What is the ChatGPT user agent?
ChatGPT-User. It's used when a ChatGPT user or a custom GPT asks to load a specific page. It isn't an automatic crawler, and OpenAI says robots.txt rules may not apply to it.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls use for Gemini training and grounding in Gemini apps and Vertex AI. AI Overviews use the normal Googlebot index.
Do AI crawlers respect robots.txt?
The major search and training crawlers from OpenAI, Anthropic, Google, Apple and Perplexity say they do. User-triggered fetchers may not, and some smaller or less transparent bots are reported to ignore it. Use a firewall or CDN rule for bots that don't comply.
Is llms.txt needed for AI crawlers?
No major AI search provider has confirmed using llms.txt, and Google has said it isn't needed. robots.txt is what controls access.
See what AI crawlers see on your site
ElixRank's GEO audit checks crawler access, rendering and citability across your key pages, with your Search Console data.
Try ElixRank free →