Why your website turns AI crawlers away

What the problem is

Some websites answer automated visitors with a security check (“Checking your browser…”, a CAPTCHA, or an “access denied” page) instead of the real page. This usually comes from a firewall or bot-protection service, such as Cloudflare, or from a security plugin.

People in a browser pass the check without noticing. Crawlers don't run the check, so all they see is the challenge page.

Why it matters for AI search

If AI crawlers only ever see a security check, they have nothing to read: no business name, no services, no address. Assistants can't describe or recommend you from your own site.

How to fix it

1. Find out what protects your site: your host's control panel, your DNS provider (often Cloudflare), or a security plugin.
2. Look for settings that block bots or AI crawlers, or that challenge all automated traffic. Allow verified search and AI crawlers through, and keep the protection for everything else.
3. If someone else manages your website or hosting, send them this page and ask them to allow verified crawlers.

On your site builder

Menus change from time to time. If a setting has moved, your site builder's help pages will have the current steps.

How to check it's fixed

• Ask whoever made the change to confirm that verified crawlers now get your normal page.
• Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The “Site reachable by bots” score shows whether we got your page or a security check.