Skip to content
bogl.ai

How to let AI crawlers read your website (robots.txt)

What the problem is

Every website can have a robots.txt file, at /robots.txt (for example yourbusiness.co.uk/robots.txt). It tells crawlers which parts of the site they may read. Each AI company uses its own named crawlers, such as OAI-SearchBot and GPTBot (OpenAI), Claude-SearchBot and ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Bingbot.

If robots.txt has a Disallow: / rule for one of these names, or for User-agent: * (every crawler), that crawler stays away from the whole site.

A robots.txt that returns a server error (a 5xx status) is a problem too. Google, for one, documents that it treats the whole site as off-limits while its robots.txt returns a server error.

Why it matters for AI search

AI assistants that search the web read pages through these crawlers. A blocked crawler can't fetch your pages, so that assistant can't describe or recommend you from your own website, and it may rely on what other sites say about you instead.

Some crawlers collect training data (for example GPTBot, ClaudeBot, CCBot) and others fetch pages to answer a search (for example OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking training crawlers is a reasonable choice for some businesses. Blocking the search crawlers means you're left out of those assistants' answers.

How to fix it

  1. Open your robots.txt in a browser: your web address followed by /robots.txt.
  2. Look for a User-agent: line naming an AI crawler (or *) followed by Disallow: /. That pair blocks the whole site.
  3. Remove those lines, or change Disallow: / to Allow: / for the crawlers you want to let in. Leave rules for private areas (such as /admin or /checkout) as they are.
  4. If robots.txt doesn't load at all, or shows an error page, ask your host to make it load normally. A file containing just the two lines below is enough.
User-agent: *
Allow: /

On your site builder

Wix

  • In your Wix dashboard, open the SEO settings (Marketing & SEO, or SEO & GEO on newer dashboards) and use the Robots.txt Editor. Remove the Disallow lines for the crawlers you want to allow, then save.

Squarespace

  • Squarespace doesn't let you edit robots.txt directly. In Settings, find Crawlers: if the option to block known AI crawlers is on, turn it off. The option to block search engine crawlers should also be off.

WordPress

  • WordPress makes a robots.txt for you. If you use an SEO plugin, edit it there: Yoast SEO has it under Tools → File editor, and Rank Math under General Settings → Edit robots.txt.
  • If your site has a real robots.txt file in its main folder, that file is used instead: edit it with your host's file manager.
  • Also check that Settings → Reading → “Discourage search engines from indexing this site” is unticked.

Shopify

  • Shopify's robots.txt comes from the theme. If it has been customised, the rules are in the robots.txt.liquid template: Online Store → Themes → … → Edit code, then look under Templates.

Menus change from time to time. If a setting has moved, your site builder's help pages will have the current steps.

How to check it's fixed

  • Open /robots.txt again and make sure it loads and no longer blocks the crawlers you want.
  • Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The crawler table on your result shows each one as Allowed or Blocked.

Check your website

The free check shows your score out of 100 and your top 3 issues in about 10 seconds. No sign-up.