Free AI robots.txt checker: which AI crawlers can read your site?

What robots.txt is

Robots.txt is a small text file at the top level of your website, at /robots.txt after your web address. A crawler is a program that visits websites automatically and reads their pages. The file tells each crawler, by name, which parts of your site it may visit.

The official standard describes it as a way for site owners “to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.”

AI search crawlers and AI training crawlers

AI companies run different crawlers for different jobs. It helps to tell the two main kinds apart.

Search crawlers read your pages so an AI tool can show and link your site when people search:

• OAI-SearchBot (OpenAI). OpenAI says it “is used to surface websites in search results in ChatGPT’s search features.”
• Claude-SearchBot (Anthropic). Anthropic says blocking it “may reduce your site’s visibility and accuracy in user search results.”
• PerplexityBot (Perplexity). Perplexity says it “is designed to surface and link websites in search results on Perplexity.”

Training crawlers collect content that may be used to train AI models:

• GPTBot (OpenAI). Blocking it tells OpenAI your content should not be used to train its AI models.
• ClaudeBot (Anthropic). Blocking it tells Anthropic to leave your future content out of its training data.

Each rule is separate. OpenAI gives the example of allowing OAI-SearchBot so you can appear in search results, while blocking GPTBot so your content is not used for training. You can choose for each crawler.

Google-Extended (Google) is different. It is not a separate crawler but a name Google calls a “standalone product token”. Google says it controls whether your content may be used to train Gemini models, and that it does not affect whether your site appears in Google Search. Google’s own page describes everything it covers. Read it before you block Google-Extended: Google’s page on its common crawlers.

There is a third kind too. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the AI tool about it. Because a person asked for the page, OpenAI says robots.txt rules “may not apply”, and Perplexity says its fetcher “generally ignores robots.txt rules”.

What “Blocked” means

“Blocked” means your robots.txt has a rule asking that crawler not to read your site. Blocking a training crawler can be a sensible choice. Blocking a search crawler matters more. For example, OpenAI says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers”, though they can still appear as plain links.

Sometimes a block is a mistake. A general rule meant for other bots, or an old setting left by a web designer or plugin, can block a crawler you wanted to let in.

How to change it, in general

To change a result, edit the robots.txt file at the top level of your site. Find the rule for that crawler’s name and remove or change its “Disallow” line. If you have subdomains (separate addresses with a different word before your main web address, such as one for an online shop), each one has its own file. Anthropic says blocking IP addresses is not a reliable way to opt out. The change needs to be made in robots.txt.

Changes are not always instant. OpenAI says its search systems can take about 24 hours to adjust after you update robots.txt.

The full report costs £39, one-off, and includes ready-made fixes such as robots.txt lines.

Robots.txt is a request, not a lock

Robots.txt asks crawlers to behave in a certain way. It does not force them. The official standard says the rules “are not a form of access authorization”. So do not rely on it to protect private pages.

Many large crawlers say they follow it. Anthropic says its bots respect robots.txt. Google says its common crawlers always obey robots.txt rules when they crawl automatically. As noted above, fetches that a person asks for may be treated differently.

Check your whole homepage too

Robots.txt is only one part of being readable by Google and AI. The free bogl.ai check looks at robots.txt rules for each AI crawler, plus things like bot protection, HTTPS and the sitemap. It gives a score out of 100 and the top three issues for your homepage in about 10 seconds, with no sign-up.

Run the free website check