# bogl.ai fix guides: making a website readable by AI search > Plain-English guides from bogl.ai (Aingaran Limited, London) to the problems its AI Visibility Check finds most often: what each problem is, why it matters for ChatGPT, Claude, Perplexity, Google's AI and Copilot, and how to fix it on Wix, Squarespace, WordPress and Shopify. Free check: https://www.bogl.ai/ ## How to let AI crawlers read your website (robots.txt) Source: https://www.bogl.ai/fix/allow-ai-crawlers Your robots.txt can block ChatGPT, Claude, Perplexity and other AI crawlers. How to find the rules that block them and remove them, on any site builder. ### What the problem is Every website can have a robots.txt file, at `/robots.txt` (for example `yourbusiness.co.uk/robots.txt`). It tells crawlers which parts of the site they may read. Each AI company uses its own named crawlers, such as `OAI-SearchBot` and `GPTBot` (OpenAI), `Claude-SearchBot` and `ClaudeBot` (Anthropic), `PerplexityBot`, `Google-Extended` and `Bingbot`. If robots.txt has a `Disallow: /` rule for one of these names, or for `User-agent: *` (every crawler), that crawler stays away from the whole site. A robots.txt that returns a server error (a 5xx status) is a problem too. Google, for one, documents that it treats the whole site as off-limits while its robots.txt returns a server error. ### Why it matters for AI search AI assistants that search the web read pages through these crawlers. A blocked crawler can't fetch your pages, so that assistant can't describe or recommend you from your own website, and it may rely on what other sites say about you instead. Some crawlers collect training data (for example `GPTBot`, `ClaudeBot`, `CCBot`) and others fetch pages to answer a search (for example `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`). Blocking training crawlers is a reasonable choice for some businesses. Blocking the search crawlers means you're left out of those assistants' answers. ### How to fix it 1. Open your robots.txt in a browser: your web address followed by `/robots.txt`. 2. Look for a `User-agent:` line naming an AI crawler (or `*`) followed by `Disallow: /`. That pair blocks the whole site. 3. Remove those lines, or change `Disallow: /` to `Allow: /` for the crawlers you want to let in. Leave rules for private areas (such as `/admin` or `/checkout`) as they are. 4. If robots.txt doesn't load at all, or shows an error page, ask your host to make it load normally. A file containing just the two lines below is enough. Example: ``` User-agent: * Allow: / ``` ### On Wix - In your Wix dashboard, go to SEO & GEO, then Tools and settings → Robots.txt Editor. Remove the Disallow lines for the AI crawlers you want to allow, and save. Reset to Default puts back Wix's standard file. - Wix's own help article on blocking AI crawlers suggests adding Disallow lines for crawlers such as GPTBot and CCBot, so check whether someone followed it. ### On Squarespace - Squarespace doesn't let you edit robots.txt. In Settings → Crawlers, turn off “Block known artificial intelligence crawlers” (and “Block Search engine crawlers”). That setting doesn't cover every AI crawler, so some blocks can't be changed on Squarespace. ### On WordPress - On WordPress.com, check Settings → Reading → “Prevent third-party sharing”: it adds AI crawlers to your robots.txt. - On your own WordPress site, edit robots.txt in your SEO plugin: Yoast SEO under Tools → File editor; Rank Math under General Settings → Edit robots.txt (Advanced Mode). A real `robots.txt` file in the main folder takes priority over both. ### On Shopify - Shopify's robots.txt only changes if your theme has a `robots.txt.liquid` template (Online Store → Themes → … → Edit code). If it does, remove the rules blocking AI crawlers; deleting the template puts back Shopify's default. Duplicate your theme before editing code. ### How to check it worked - Open `/robots.txt` again and make sure it loads and no longer blocks the crawlers you want. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The crawler table on your result shows each one as Allowed or Blocked. ## Why your website turns AI crawlers away Source: https://www.bogl.ai/fix/let-crawlers-past-bot-protection Firewalls and bot protection can show AI crawlers a security check instead of your page. How to allow verified crawlers through. ### What the problem is Some websites answer automated visitors with a security check (“Checking your browser…”, a CAPTCHA, or an “access denied” page) instead of the real page. This usually comes from a firewall or bot-protection service, such as Cloudflare, or from a security plugin. People in a browser pass the check without noticing. Crawlers don't run the check, so all they see is the challenge page. ### Why it matters for AI search If AI crawlers only ever see a security check, they have nothing to read: no business name, no services, no address. Assistants can't describe or recommend you from your own site. ### How to fix it 1. Find out what protects your site: your host's control panel, your DNS provider (often Cloudflare), or a security plugin. 2. Look for settings that block bots or AI crawlers, or that challenge all automated traffic. Allow verified search and AI crawlers through, and keep the protection for everything else. 3. If someone else manages your website or hosting, send them this page and ask them to allow verified crawlers. ### On WordPress - On WordPress.com, Defensive mode (Business and Commerce plans) shows visitors a “Checking your browser” screen; turn it off when you don't need it. On your own WordPress site, check your security plugin and your host's firewall. ### On Shopify - Shopify has its own bot protection that you can't switch off, and Shopify says search engines and AI crawlers can still index stores. If our check was turned away, it may be that protection rather than a setting of yours; check any extra firewall or app you've added. ### How to check it worked - Ask whoever made the change to confirm that verified crawlers now get your normal page. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The “Site reachable by bots” score shows whether we got your page or a security check. ## What to do when AI crawlers can't load your homepage Source: https://www.bogl.ai/fix/homepage-not-loading If your homepage times out or returns an error status to automated visitors, AI crawlers can't read it. How to find and fix the cause. ### What the problem is When we request your homepage the way a crawler does, it either didn't load at all (a timeout, a name that doesn't resolve, or a refused connection) or it returned a status other than 200, such as 403 (forbidden), 404 (not found) or 500 (server error). ### Why it matters for AI search Crawlers only read pages that load normally with status 200. If your homepage fails for them, AI tools can't learn what your business does, even if the page looks fine in your browser. ### How to fix it 1. Check the address is right, including whether it starts with `www.`, and that it opens in a private browser window. 2. If it loads for you but not for us, your hosting or firewall may be blocking automated visitors. See the guide on bot protection. 3. If it returns 404, the homepage address may have changed: set it back, or redirect the old address to the new one. 4. If it returns a 5xx error, contact your host: something is failing on the server. 5. If your domain has expired or isn't pointing at your website, renew it or fix the DNS settings with your domain provider. ### How to check it worked - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. ## How to turn on HTTPS for your website Source: https://www.bogl.ai/fix/turn-on-https Sites without HTTPS are treated as less trustworthy. How to turn on a free security certificate and send http:// visitors to https://. ### What the problem is HTTPS encrypts the connection between visitors and your website and shows the padlock in the browser's address bar. Without it, browsers mark your site as “Not secure”. ### Why it matters for AI search Browsers, search engines and AI search tools treat sites without HTTPS as less trustworthy. It's also the first thing a careful customer notices. ### How to fix it 1. Turn on a security (SSL/TLS) certificate in your hosting control panel. Most hosts include one free. 2. Redirect every `http://` address to the same `https://` address, so there's only one version of each page. 3. Update links in your site settings, Google Business Profile and social profiles to use `https://`. ### On Wix - Every Wix site gets HTTPS automatically, at no cost. If your custom domain shows as not secure, check its DNS records under Domains (allow up to 48 hours after changes), and look for `http://` links inside embedded code. ### On Squarespace - Squarespace includes a free SSL certificate. In the SSL panel, set the security preference to Secure and HSTS Secure; domains connected before October 2016 may still be set to Insecure. ### On WordPress - On WordPress.com, SSL is automatic. On your own site, turn on the certificate with your host, then use Tools → Site Health → “Update your site to use HTTPS” (WordPress 5.7 and later), and ask your host to redirect http to https. ### On Shopify - Shopify includes free certificates and sends http visitors to https. If a domain shows as not secure, check it under Settings → Domains; allow up to 48 hours after connecting it. ### How to check it worked - Open your site with `http://` at the start and make sure it switches to `https://`. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. ## How to make your text readable without JavaScript Source: https://www.bogl.ai/fix/content-readable-without-javascript Most AI crawlers don't run JavaScript. If your text only appears after scripts run, they see an almost empty page. How to fix it. ### What the problem is Some websites send an almost empty page and then build the content in the browser with JavaScript. People see the finished page. Crawlers that don't run JavaScript see only what was in the page as delivered. We count the words in your homepage as delivered, before any scripts run. Very few words means most of your text is added later by scripts. ### Why it matters for AI search Most AI crawlers read the HTML they're sent and don't run JavaScript. If your services, town and contact details only appear after scripts run, those crawlers can't see them. ### How to fix it 1. If your site was built with a JavaScript framework (such as React, Vue or Angular), ask your developer to turn on server-side rendering or pre-rendering, so each page is sent with its text already in it. 2. If you use a site builder, make sure your main text is in ordinary text blocks, not inside images, embedded widgets or animations. 3. Put the essentials in plain text on the homepage: what you do, where, and how to get in touch. ### How to check it worked - In your browser, view the page source (often Ctrl+U, or Option+Command+U on a Mac) and search for a sentence from your homepage. If it's there, crawlers can read it. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. ## How to add LocalBusiness structured data Source: https://www.bogl.ai/fix/add-business-structured-data Structured data states your business name, address, phone and hours in a form machines read reliably. How to add it on Wix, Squarespace, WordPress, Shopify. ### What the problem is Structured data is a short block of code in your page, usually in a format called JSON-LD, that states facts about your business: name, address, phone number, opening hours and links to your profiles elsewhere. It uses the shared schema.org vocabulary, so search engines and other tools read it the same way. For a local business the type is `LocalBusiness`, or a more specific type such as `Plumber` or `Restaurant`. We look for it on your homepage and check it has your name, address and phone number. The full report also checks that the name and phone number are the same across your pages. ### Why it matters for AI search Reading facts from ordinary page text is guesswork; structured data states them plainly. When your name, address and phone number are marked up the same way everywhere, tools can be surer they're describing the right business and how to reach it. If different pages give different names or numbers, an AI tool may repeat an old or wrong one. ### How to fix it 1. Decide on one exact business name, one main phone number and one address, and use them everywhere: website, Google Business Profile and directories. 2. Add a JSON-LD block like the one below to your homepage, with your own details. (The full report builds this block for you from what it finds on your site.) 3. Only include facts that are true and shown to visitors on the page as well. Example: ``` ``` ### On Wix - Wix adds basic `LocalBusiness` data from Settings → Business info, but only the name, address, logo and web address: not your phone number or opening hours. - For the full block, add it to the homepage's SEO settings: Advanced SEO → Structured Data Markup (JSON-LD only, under 7,000 characters). ### On Squarespace - Squarespace adds its own business and organisation data, which you can't edit. To add the full block, paste it into Code Injection (the homepage's own header, or the site-wide header); Code Injection needs a Core, Plus or Advanced plan. ### On WordPress - Yoast SEO's free settings cover your name, logo and social profiles but not your address or phone; Rank Math's Local SEO covers address, phone and opening hours. - Or paste the full block into the header with a plugin such as WPCode (Insert Headers and Footers). On WordPress.com, adding scripts needs a plan with plugins. ### On Shopify - Paste the block into your theme's `theme.liquid`, just before `` (Online Store → Themes → … → Edit code). Duplicate the theme first; theme updates can remove code edits, and your theme may already include some structured data. ### How to check it worked - Test the homepage with the Schema Markup Validator (validator.schema.org) or Google's Rich Results Test, and make sure the block reads without errors. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The “Business facts AI can use” score shows what we found. ## How to set page titles and descriptions for AI search Source: https://www.bogl.ai/fix/page-titles-descriptions-headings Page titles, meta descriptions and headings are what AI summaries are built from. What good ones look like and where to set them on your site builder. ### What the problem is Each page has a title (shown in the browser tab and search results), a meta description (the short summary under it), and a main heading on the page itself (the H1). We also check that the page states its language and has a canonical link (the page's preferred address). We flag a missing title or description, a title outside 10–65 characters, a description outside 50–160 characters, and pages with no main heading or more than one. ### Why it matters for AI search These are the first things search engines and AI tools read to work out what a page is about. A clear title and description that say what you do and where make it easier for them to summarise and cite the page accurately. ### How to fix it 1. Title: say what you do and where, then your business name, for example “Emergency plumber in Bristol | Smith & Sons”. 2. Description: one or two plain sentences on what you offer, where, and why someone should choose you. 3. Use one main heading (H1) per page that says what the page is about. Use smaller headings for sections. 4. Give every page its own title and description, not the same text copied across pages. ### On Wix - Set each page's title and description in its SEO basics (Pages & Menu → the page's More Actions). For the main heading, set the text's HTML tag to H1 (Edit Text → SEO & Accessibility): the H1 text style on its own isn't enough. ### On Squarespace - For the homepage, set the title format and SEO site description in the SEO settings (Home tab); the homepage has no SEO tab of its own. Other pages use the SEO tab in their page settings. Use Heading 1 for one main heading. ### On WordPress - Set the title and description in your SEO plugin's box when editing each page (Yoast SEO, Rank Math), or with Jetpack SEO on WordPress.com. Check your theme doesn't add a second H1, for example on the site title. ### On Shopify - For the homepage: Online Store → Preferences → title and meta description. Shopify recommends titles under 60 characters. - For other pages, products and collections: “Search engine listing” at the bottom of each editor → Edit. ### How to check it worked - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The “Page basics” score shows what's still missing. The full report lists every page with suggested titles. ## How to remove “noindex” from your homepage Source: https://www.bogl.ai/fix/remove-noindex A noindex tag tells search engines to leave a page out. AI search tools that rely on search indexes skip it too. How to find and remove it. ### What the problem is A robots meta tag with `noindex` (or `none`) tells search engines not to include the page in their results. It's often switched on by a “hide from search engines” setting, sometimes left on by mistake after a site was built. ### Why it matters for AI search Many AI search tools find pages through search indexes such as Google's and Bing's. A homepage marked noindex is left out of those indexes, so it's much less likely to be found or cited. ### How to fix it 1. Find the setting that hides the page or the whole site from search engines, and turn it off. 2. If a developer built the site, ask them to remove `noindex` from the homepage's robots meta tag. ### On Wix - In the dashboard, go to SEO & GEO → Tools and settings → SEO Settings and make sure “Let search engines index your site” is on. Also check the page's own SEO settings, including the robots meta tag under Advanced SEO. ### On Squarespace - Squarespace homepages don't have a “hide from search” switch, so a noindex on the homepage usually comes from Code Injection: check the site-wide header and the homepage's own header code. Also make sure the site is paid and Public. ### On WordPress - Go to Settings → Reading and untick “Discourage search engines from indexing this site”. In Yoast SEO or Rank Math, check the homepage's Advanced settings allow it in search results; on WordPress.com, check “Hide page from search engines” is off. ### On Shopify - Shopify's “hide from search engines” setting only covers products, pages and blog posts. A noindex on the homepage usually comes from theme code, an app, or the store still being in private mode (Online Store → Preferences). ### How to check it worked - View the homepage's source and search for `noindex`. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. ## How to add a sitemap to your website Source: https://www.bogl.ai/fix/add-a-sitemap A sitemap lists every page on your site so crawlers don't miss any. Most site builders make one for you. How to find it, turn it on and list it in robots.txt. ### What the problem is A sitemap is a file, usually at `/sitemap.xml`, that lists the pages on your website. We look for one there and in your robots.txt. ### Why it matters for AI search Crawlers find pages by following links. A sitemap makes sure they know about every page, including ones that aren't well linked, such as individual service or location pages. ### How to fix it 1. Check whether you already have one: open your web address followed by `/sitemap.xml` (WordPress's built-in one is at `/wp-sitemap.xml`). 2. If you don't, turn it on in your site builder or SEO plugin, or ask your developer to add one. 3. Add a line to your robots.txt pointing to it, like the one below, so every crawler can find it. 4. Submit it in Google Search Console and Bing Webmaster Tools. Example: ``` Sitemap: https://www.yourbusiness.co.uk/sitemap.xml ``` ### On Wix - Every Wix site has a sitemap at `/sitemap.xml` that updates automatically. Pages you've set not to be indexed are left out. ### On Squarespace - Squarespace creates a sitemap at `/sitemap.xml`, but not for trial, private or password-protected sites. ### On WordPress - WordPress creates one at `/wp-sitemap.xml`, but not while “Discourage search engines” is ticked. SEO plugins replace it with their own (Yoast SEO's is at `/sitemap_index.xml`). WordPress.com sites have one at `/sitemap.xml`. ### On Shopify - Shopify creates a sitemap at `/sitemap.xml`, but crawlers can't read it while the store is in private mode. ### How to check it worked - Open `/sitemap.xml` (or your plugin's sitemap address) and make sure it lists your pages. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. ## How to get your website into Common Crawl Source: https://www.bogl.ai/fix/get-into-common-crawl Common Crawl is a free public web archive that many AI models are trained on. Why your site may be missing and what you can do about it. ### What the problem is Common Crawl is a non-profit that crawls the web and publishes the results as a free archive, a new crawl every month or two. Many AI models are trained partly on it. The full report checks whether any of your pages are in the latest crawl. ### Why it matters for AI search If none of your pages are in it, AI models trained on it are less likely to know your business exists from your own website. You can't submit a site to Common Crawl. Its crawler, `CCBot`, finds sites by following links from other sites. ### How to fix it 1. Make sure `CCBot` isn't blocked in your robots.txt (see the robots.txt guide). 2. Get links from other websites: local directories, trade bodies, suppliers, local news and community sites. Each one is a route for crawlers to find you. 3. Keep your homepage and main pages reachable without logins, pop-ups that block content, or bot protection that challenges crawlers. 4. Check again after the next crawl is published. It can take a few crawls for a site to appear. ### How to check it worked - Common Crawl publishes an index you can search at index.commoncrawl.org. A new full report also checks the latest crawl. ## How to answer customer questions for AI search Source: https://www.bogl.ai/fix/answer-customer-questions People ask AI assistants questions. Pages that ask and answer real customer questions in plain words are easier to quote. How to write an FAQ that helps. ### What the problem is We look for questions answered in your own words: headings that are questions, or an FAQ marked up with `FAQPage` structured data. The full report flags sites with no FAQ and fewer than three question headings. ### Why it matters for AI search People ask assistants full questions, such as “who fixes boilers in Bristol on a Sunday?”. A page that asks and answers that kind of question plainly is easier for an assistant to find and quote. ### How to fix it 1. Write down 5–10 questions customers actually ask you by phone, email or in person. 2. Add them to a page as headings, each followed by a short, specific answer: prices or price ranges, areas you cover, opening times, how quickly you can come out. 3. Keep the answers as plain text on the page, not hidden in images or only inside collapsed sections that need a script to load. 4. Optionally, mark the questions up with `FAQPage` structured data. Only mark up questions and answers that are visible on the page. ### How to check it worked - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The full report checks question headings and FAQ markup across up to 15 pages. ## Why your website needs a contact page Source: https://www.bogl.ai/fix/add-a-contact-page A clear contact page with your address, phone number and opening hours is where AI tools look for how to reach you. What to include. ### What the problem is The full report looks for a page whose address or link mentions “contact”. If it can't find one, it flags it. ### Why it matters for AI search A contact page is the usual place for a business's address, phone number, email and opening hours. Search engines, AI tools and customers all look for it there. ### How to fix it 1. Add a page called Contact, at an address like `/contact`, and link to it from your main menu and footer on every page. 2. Put your full details on it as text: business name, address (or the areas you cover), phone number, email and opening hours. 3. Use exactly the same name, address and phone number as on your homepage, Google Business Profile and directory listings. 4. A contact form is fine as well, but not instead of the details: crawlers can't read what's behind a form. ### How to check it worked - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow. The full report checks for a contact page. ## How to show up when customers ask AI Source: https://www.bogl.ai/fix/show-up-in-ai-search If an AI assistant searching for what you do doesn't find your site, it recommends someone else. The steps that make a difference, in order. ### What the problem is In the full report we have an AI assistant search the web the way a customer would, for example “emergency plumber in Bristol”, and note whether your website comes up. This finding means it didn't, or only rarely. ### Why it matters for AI search When an assistant answers a question like that, it can only recommend businesses it finds. If your site isn't in what it finds, it recommends someone else. No one can promise a place in AI answers, and results change from day to day. The steps below make it more likely, and they help ordinary search as well. ### How to fix it 1. Fix the access problems first: AI crawlers allowed in robots.txt, no bot protection turning them away, and text readable without JavaScript. 2. Say what you do and where in plain words, in your page titles, main headings and opening paragraph: “Boiler repair in Bristol”, not just “Welcome”. 3. Give each main service, and each main town you cover, its own page. 4. Add LocalBusiness structured data and a contact page with consistent details. 5. Claim and complete your Google Business Profile, and list your business on Bing Places and reputable local and trade directories with the same name, address and phone number. 6. Ask happy customers for reviews on the platforms your customers use. ### How to check it worked - Changes take time to be picked up: search indexes and assistants revisit sites on their own schedules. Run the free check again after making changes, and a new full report after a few weeks. ## What is llms.txt, and how to add one to your website Source: https://www.bogl.ai/fix/add-llms-txt llms.txt is a short plain-text summary of your website for AI tools. What it is, what to put in it, and how to add one. ### What the problem is llms.txt is a proposed standard (llmstxt.org): a plain-text file at `/llms.txt` that summarises what your website is about and links to its most useful pages, written in Markdown. It's for AI tools that want a quick, clean overview of a site. It's a proposal, not a rule every AI tool follows, so it carries little weight in our score. It's quick to add and does no harm. ### Why it matters for AI search A short, accurate summary in your own words gives AI tools that read it a clear statement of what you do, where, and which pages matter. ### How to fix it 1. Write a file like the example below: your business name as a heading, a one-line summary, then a list of your key pages with a few words on each. 2. Upload it so it opens at your web address followed by `/llms.txt`, as plain text. 3. Keep it up to date when your services or pages change. (The full report includes a draft built from your site.) Example: ``` # Smith & Sons Plumbing > Emergency and planned plumbing in Bristol and North Somerset since 1998. Gas Safe registered. ## Pages - [Services](https://www.smithandsons.co.uk/services): boiler repair, bathrooms, leaks - [Areas we cover](https://www.smithandsons.co.uk/areas): Bristol, Clevedon, Portishead - [Contact](https://www.smithandsons.co.uk/contact): phone, email, opening hours ``` ### On Wix - Wix creates an llms.txt for sites on a premium plan with a connected domain and indexing on. View or edit it under SEO & GEO → Tools and settings; once you edit it, it stops updating automatically until you reset it. ### On WordPress - Upload the file to your site's main folder (ask your host if you're not sure how), or use a plugin that serves an llms.txt. On WordPress.com, uploading to the main folder needs a Business plan or higher. ### How to check it worked - Open `/llms.txt` on your site and make sure it shows your text, not an error page. - Run the free check on bogl.ai again. Results are kept for 24 hours, so if you checked the same site earlier today you'll see the earlier result; check again tomorrow.