There are three kinds of AI bot. Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot) collect pages that may be used to train future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) build the indexes that AI assistants search when they answer. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked.
A business that wants to be recommended should allow every search crawler and fetcher, and make a deliberate choice about training crawlers. Then check the firewall, because a CDN block overrides robots.txt. Our free AI crawler checker tests your site against the list below.
The list: AI crawlers and user agents in 2026
Every row except Bytespider comes from the company's own documentation, checked on 21 and 22 October 2026, and each token links to that page. "Token" is the name you write after User-agent: in robots.txt.
| Token | Company | Purpose | Follows robots.txt? |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search index for ChatGPT search | Yes. Opted-out sites "will not be shown in ChatGPT search answers" |
| GPTBot | OpenAI | Training | Yes |
| ChatGPT-User | OpenAI | User-triggered fetch | "robots.txt rules may not apply" |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ChatGPT ads | Only visits pages submitted as ads; not used for training |
| Claude-SearchBot | Anthropic | Search index for Claude | Yes |
| ClaudeBot | Anthropic | Training | Yes, and supports Crawl-delay |
| Claude-User | Anthropic | User-triggered fetch | Yes. Blocking it stops Claude retrieving your page for users |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | Token only, no crawler: Gemini training and grounding in Gemini Apps | Yes (it is a robots.txt control) | |
| Google-CloudVertexBot | Crawls a site owner asked for, to build Vertex AI agents | Yes; Googlebot rules also apply to it | |
| GoogleOther | One-off crawls for research and development | Yes | |
| Google-Agent | Agents on Google infrastructure acting on a user's request | User-triggered fetchers "generally ignore robots.txt rules" | |
| PerplexityBot | Perplexity | Search index; "not used to crawl content for AI foundation models" | Yes |
| Perplexity-User | Perplexity | User-triggered fetch | "generally ignores robots.txt rules" |
| Bingbot | Microsoft | Bing search, Copilot and Bing's grounding API | Yes. Use NOARCHIVE or NOCACHE tags to limit Copilot use |
| Applebot | Apple | Search in Siri, Spotlight and Safari; data may also train Apple models | Yes; follows Googlebot rules if Applebot is not named |
| Applebot-Extended | Apple | Token only: opt out of training Apple's foundation models | Yes. It "does not crawl webpages" |
| Meta-ExternalAgent | Meta | Training AI models and indexing content | Yes |
| Meta-WebIndexer | Meta | Search for Meta AI; helps Meta "cite and link to your content" | Yes |
| Meta-ExternalFetcher | Meta | User-triggered fetch and agent tasks | "may bypass robots.txt rules" |
| Amazonbot | Amazon | Improving Amazon products; "may be used to train Amazon AI models" | Yes; no Crawl-delay |
| Amzn-SearchBot | Amazon | Search in Amazon products such as Alexa; no training | Yes |
| Amzn-User | Amazon | User-triggered fetch, for example live answers in Alexa | "may not follow all robots.txt directives" |
| CCBot | Common Crawl | Open web archive, widely used to train AI models | Yes |
| MistralAI-Index | Mistral AI | Search index; "not used for generative AI training of any kind" | Yes |
| MistralAI-User | Mistral AI | User-triggered fetch in Mistral's Vibe assistant | Yes; it "governs which sites these user requests can be made to" |
| MistralAI-Training | Mistral AI | Training | Yes |
| DuckAssistBot | DuckDuckGo | Real-time fetches for DuckDuckGo's AI answers; not used for training | Yes; changes take effect after 72 hours |
| Bytespider | ByteDance | No official documentation found | Unknown |
Most of these companies also publish the IP addresses their bots use, linked from the same documentation pages (for example openai.com/searchbot.json, perplexity.com/perplexitybot.json and bing.com/toolbox/bingbot.json). Use those lists to tell a real bot from a fake one that only borrows its name. Anthropic warns against the reverse: blocking its IP addresses to opt out "may not work correctly", because it stops the bot reading your robots.txt.
Training, search and user fetch: what each choice costs you
Block a training crawler
Your future pages should be left out of that company's training data. You stay in its search answers. The cost: future models may know less about you, so answers from memory may leave you out.
Block a search crawler
You drop out of that assistant's index. OpenAI says such sites "will not be shown in ChatGPT search answers". For a business that wants buyers, this is almost always a mistake.
Block a user fetcher
When a buyer asks about you, or pastes your link, the assistant may not be able to open your page. Some fetchers ignore robots.txt anyway, so the block may not even work.
Two Google details trip people up. Google-Extended does not touch Google Search: to leave AI Overviews and AI Mode you use the Search generative AI control in Search Console instead, and Google says that control "doesn't affect AI training". And because Google-Extended has no user agent of its own, you will never see it in your logs.
The robots.txt we recommend for a business that wants to be recommended
Keep it short. Under the robots.txt standard (RFC 9309), a crawler that finds a group with its own name obeys that group only. It ignores the User-agent: * group completely. So naming bots just to "allow" them is risky: they then skip every rule in your general group. The safest file lets every bot follow one general group, and names only the bots you want to keep out.
# Everyone: crawl the site, skip private areas
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
That is the whole file for most businesses. Every search crawler, fetcher and training crawler that follows robots.txt is allowed in, except into the private folders.
If you object to AI training, add one group for the training tokens. You stay in AI search answers:
# Optional: opt out of AI training, stay in AI search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /
Think twice before adding Google-Extended to that group. Google says it also controls "grounding" in Gemini Apps, so blocking it can keep you out of Gemini app answers that cite the web. Amazonbot is left out on purpose, because Amazon describes it as both product improvement and training.
Some sites also add Cloudflare's Content Signals line, such as Content-Signal: search=yes, ai-train=no. Cloudflare itself says content signals "express preferences; they are not technical countermeasures", and we found no AI company documenting that it reads them. The IETF's AI Preferences working group is drafting a standard way to say the same thing, but it is not finished.
Six mistakes we check on every site
- The firewall says no. Since 1 July 2025 Cloudflare has asked every new domain whether to allow AI crawlers, with blocking as the default. Bot-fight modes and security plugins can do the same. robots.txt cannot override a firewall.
- robots.txt returns a server error. RFC 9309 says a 5xx response means crawlers "MUST assume complete disallow". One broken file blocks every bot.
- Copy-paste blocklists. A "block all AI bots" list can include search crawlers such as OAI-SearchBot, PerplexityBot or Claude-SearchBot, which removes you from AI search, not just training. Read every line before you paste one.
- Named groups that skip your rules. As above: a bot with its own group ignores
User-agent: *. - Waiting for instant results. OpenAI and Perplexity say changes take about 24 hours, Amazon may use a copy up to 30 days old, DuckDuckGo says 72 hours, and RFC 9309 lets crawlers cache the file for up to 24 hours.
- Trusting the name in the logs. Anyone can send a request that says "GPTBot". Check the IP against the published list before you act on it.
What about bots that ignore the rules?
robots.txt is a request, not a lock. The standard says its rules "are not a form of access authorization". Disputes happen. On 4 August 2025 Cloudflare said Perplexity used undeclared crawlers to get around no-crawl rules, and removed it from its verified bot list. Perplexity replied that Cloudflare had confused its traffic with requests from an outside browser service. We can't settle that argument, but the lesson is practical: if you truly need to keep a bot out, use a firewall rule. If you want to be recommended, the bigger risk is blocking the bots you need.
What we'd do for you
Every engagement starts by checking access: robots.txt, firewall and CDN rules, server logs for the real bots, and the pages each assistant can actually read. The free AI check asks 10 of your buyers' questions on ChatGPT and Gemini (Google AI). The audit (AED 1,950, or USD 549 for clients outside the GCC at adjoltz.com/us) adds a full crawler and access review across four AI tools. Monthly plans (Foundation AED 4,500 or USD 1,249, Growth AED 8,500 or USD 2,349, Authority AED 16,000 or USD 4,399) run month to month, no setup fee, with an unconditional 30-day money-back guarantee (rules). For a quick self-check, run your site through our AI crawler checker.
What we don't promise
- That allowing crawlers gets you recommended. Access is the entry ticket; pages and mentions do the work.
- That this list stays complete. Companies add and rename bots often: since 2025, OpenAI, Amazon, Meta, Mistral and Google have all added new ones. We re-check the official pages and date every update.
- That robots.txt can keep out a bot that chooses to ignore it.