robots.txt is a small text file at the root of your site that tells crawlers what they may read. This checker downloads yours, applies the matching rules of the robots.txt standard (RFC 9309), and gives 28 AI crawlers a verdict: allowed, partly blocked or blocked.
The verdict that matters most is AI search. If OAI-SearchBot is blocked, OpenAI says your site will not be shown in ChatGPT search answers. Blocking a training crawler such as GPTBot is a separate choice and does not remove you from AI search.
Three kinds of AI crawler
AI companies run more than one crawler, and each does a different job. The checker labels every bot with one of these:
| Kind | What it does | Examples | If you block it |
|---|---|---|---|
| AI search | Finds pages to show and link inside AI answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | You can disappear from those answers |
| User request | Opens a page because a person asked the assistant to | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | Some vendors say these may not follow robots.txt, because a human asked |
| Training | Collects pages that may be used to train future models | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended | Your pages should not be used in training. AI search is not affected |
Two names on the list are not crawlers at all. Google-Extended and Applebot-Extended are switches. Google says Google-Extended controls whether your content is used to train Gemini models and to ground answers in Gemini apps and Vertex AI, and that it does not affect your inclusion in Google Search. Apple says Applebot-Extended does not crawl pages; it only decides whether what Applebot collected may train Apple's models.
Google AI Overviews and AI Mode work differently. Google says the control for those is your robots.txt rules for Googlebot, plus page tags such as nosnippet and noindex. Since October 2026, Google's documentation also points to a site-wide Search generative AI control in Search Console. That setting lives in your Search Console account, not in robots.txt, so no outside tool can see it.
How the checker reads your robots.txt
We follow the same rules the crawlers follow, so the verdict matches what they will do:
- Groups. A group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler obeys the group that names it, matched without caring about capital letters. If several groups name it, their rules are combined.
- The star group. Only when no group names a crawler does it fall back to the rules under
User-agent: *. So a site can block everyone with the star group and still let OAI-SearchBot in with its own group. - Longest rule wins. For any address, the rule with the longest matching path decides. If an Allow and a Disallow are the same length, Allow wins.
*matches any run of characters and$marks the end of the address. - No file, or a broken one. If robots.txt is missing (a 404), every crawler may read everything. If it answers with a server error or does not answer, the standard says crawlers must treat the whole site as blocked. That is why a slow or failing robots.txt is worse than none.
- Apple's fallback. Apple says that when your file does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. The checker applies that.
"Partly blocked" means the homepage and most pages are open, but at least one ordinary content address is closed to that crawler, such as /blog/, /products/ or /about. We test a fixed list of typical content addresses for this. Rules that close only private areas like /admin or /cart do not count against you, and we list them so you can see them.
How sites block AI search by accident
Four ways a site can block AI search without meaning to:
- A copied block list. Some lists of "AI bots to block" shared online mix training crawlers with search crawlers. Paste one in and you may block OAI-SearchBot or PerplexityBot while trying to stop training.
- A CDN setting. Cloudflare can add a managed block to the top of your robots.txt. Its documentation shows that block disallowing Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, with the default content signal
search=yes, ai-train=no, use=reference. Those are training crawlers, so search is usually fine, but check what your version says. The checker flags Cloudflare's markers when it sees them. - A firewall that ignores the file. Cloudflare's own documentation says robots.txt "compliance is voluntary" and points to AI Crawl Control for actual blocking. The reverse also happens: a firewall rule can block a crawler your robots.txt allows. If our check was challenged or refused, we say so, because AI crawlers may hit the same wall.
- One line too many.
Disallow: /underUser-agent: *, left over from a staging site, closes everything to every crawler that has no group of its own.
What we saw when we ran it
When we ran this checker on 22 October 2026:
- nytimes.com blocks 19 of the 28 crawlers, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot and Google-Extended, while Googlebot and Bingbot stay open. That looks deliberate: it keeps the paper in Google and Bing search and out of most AI tools.
- adjoltz.com allows all 28, with only private paths such as /api/ closed. setupcollective.ae (a client) allows all 28 with no paths closed.
Your result is only as current as your robots.txt. We cache each result for 6 hours, so a fix you make now shows up after that.
Using the fix
Under every result you get paste-ready lines. They give each AI search and user-request crawler its own group that allows the site, and they copy your existing private closures (the Disallow rules under the star group) into each group, so opening a crawler never opens your admin or checkout pages. Tick the box if you also want to block the training crawlers.
- Open your robots.txt (in WordPress, Shopify and most hosts it is a setting or a file in the site root).
- Delete any old groups for the same crawlers, because rules for one crawler combine across groups.
- Paste the lines at the top, keep your other rules, and save.
- Check your CDN or firewall bot settings, then run this checker again after a few hours. OpenAI says its systems can take about 24 hours to notice a robots.txt change, and Meta says up to 24 hours.
What this tool cannot tell you
- Whether a crawler actually visits. Only your server logs show that.
- Settings that live in an account, such as Google's Search generative AI control or a firewall rule we were not challenged by.
- Whether AI recommends you. Being readable is the entry ticket, not the prize. ChatGPT and Gemini still choose from many readable sites. To see what they say about your business today, get the free AI check.