SEO check for my website: what a good check looks at for Google and AI

If you run a small business and want an “SEO check” for your website, it helps to know what a good check looks at. This guide covers the basics for Google and for AI assistants. For each one, it explains what it is, why it matters and how to spot a problem.

First, a word on expectations. Google itself says: “There are no secrets here that’ll automatically rank your site first in Google (sorry!).” SEO (search engine optimisation) helps search engines understand your pages. It is not a trick. Changes also take time to show. Google says some take a few hours and others can take several months.

A quick first check: is Google finding your site?

Before anything else, see whether Google knows your site exists. Search Google for site: followed by your web address, for example site:yourbusiness.co.uk. If few or none of your pages come up, start with the sections on noindex and robots.txt below.

Page titles

What it is. The page title is the name of a page, set in a part of the code called the title element. It is usually the blue link people see in Google. Most website builders turn the title you type into this element for you.

Why it matters. Google says the title link is often the main thing people use to decide which result to click. Google says a good title is unique to the page, clear and concise, and accurately describes what is on the page.

How to spot a problem. Look for:

• Pages with no title, or a vague one such as “Home”.
• The same title on lots of pages, which makes them hard to tell apart.
• Titles that are half-empty, out of date or wrong.
• Titles stuffed with keywords. Google says to avoid this.

If your titles are weak, Google may rewrite them in its results.

Meta descriptions

What it is. A meta description is a short summary of a page, usually one or two sentences. It sits in the page code and visitors don’t see it on the page.

Why it matters. Google mostly builds the snippet (the short text under the link) from your page content. But it may use your meta description when it describes the page better. Google says a good one is short, unique to one page and covers the most relevant points.

How to spot a problem. Watch for the same description on every page, or no description at all. If you can’t write one for every page, Google suggests at least covering your home page and your most popular pages. A description that is just a list of keywords is less likely to be shown. Google also doesn’t use the old “meta keywords” tag, so you don’t need to fill it in.

Many website builders have a search engine settings area where you can write titles and descriptions without touching any code.

Headings

What it is. Headings are the titles and subtitles on a page. The main one is called the h1.

Why it matters. Headings break up long text and help visitors find their way around a page. Google may also use headings such as the h1 when it writes the title link for your page. Google suggests making your main title stand out, for example by putting it in the first visible h1.

How to spot a problem. Check that each page has one clear main heading that says what the page is about. Several headings that look equally important can make it unclear which is the main title. Google says there is no ideal number of headings. It also says heading order doesn’t matter for Google Search, although a logical order is good for screen readers, which some visitors use.

Being indexable (noindex)

What it is. Being “indexed” means Google has stored your page and can show it in results. A noindex rule tells search engines not to show a page. It can sit in the page code (a meta tag) or be sent by the server (an HTTP header).

Why it matters. Google says that with noindex it will drop the page entirely from its results, even if other sites link to it. So if a noindex sits on a page you want people to find, that page won’t show in Google. Other search engines may treat noindex a bit differently.

How to spot a problem. Look for noindex on pages you want found. It also works the other way. If you’ve added noindex to a page and it still shows in Google, Google may not have visited the page again yet. Google Search Console is Google’s free tool for site owners. Its URL Inspection Tool shows how Google sees a page. Its Page Indexing report lists the pages where Google found a noindex rule.

Mobile-friendly pages

What it is. A mobile-friendly page works well on a phone. Google uses the mobile version of your site for indexing and ranking. This is called mobile-first indexing.

Why it matters. Whatever is on your mobile pages is what Google mainly goes by. Google doesn’t strictly need a mobile version to include your site, but it very strongly recommends having one. It recommends responsive design, where one page adapts to any screen size, because it is the easiest to set up and maintain.

How to spot a problem. Check that your mobile pages:

• Have the same content as your desktop pages, including headings, titles and descriptions.
• Don’t carry a noindex that the desktop version doesn’t have.
• Don’t hide important text behind swipes, clicks or typing. Google won’t load content that needs someone to interact with the page first.

If your website theme can’t be made to work on phones, you may need a new mobile-friendly theme.

HTTPS

What it is. HTTPS is the secure version of a web address, the one that starts with “https” rather than plain “http”.

How to spot a problem. Look at your web address in the browser and check it starts with “https”. It is one of the basic things a website check should look at, and bogl.ai’s free check includes it.

robots.txt

What it is. robots.txt is a small text file at the root of your site, for example yourbusiness.co.uk/robots.txt. It tells crawlers which parts of your site they may visit. Crawlers are the automated programs that read websites for search engines and AI tools.

Why it matters. robots.txt controls crawling (visiting), not indexing (showing). Google says robots.txt can stop it crawling your pages but may not always stop them being indexed. A page blocked by robots.txt can still turn up in results, because Google never gets to see its noindex rule. Google also doesn’t support putting noindex in robots.txt.

How to spot a problem. Open your robots.txt and check it isn’t blocking pages you want found. If you want a page kept out of Google, use noindex and let Google crawl the page so it can see the rule.

AI crawlers

What it is. AI companies run their own crawlers, and you can name each one in robots.txt. OpenAI, the company behind ChatGPT, says it uses OAI-SearchBot and GPTBot “to enable webmasters to manage how their sites and content work with AI.” Each one has its own setting.

Why it matters. These crawlers do different jobs, so blocking one is not the same as blocking another:

• OAI-SearchBot is used for ChatGPT search. OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing it.
• GPTBot collects content that may be used to train AI models. OpenAI says blocking GPTBot signals that your content should not be used for training.
• ChatGPT-User visits pages when a person asks ChatGPT to do something. Because a user starts these visits, OpenAI says robots.txt rules may not apply.

Allowing a crawler does not mean an AI assistant will mention your business. But blocking the wrong one by accident can keep you out of places you may want to appear.

How to spot a problem. Read your robots.txt line by line. Look for rules that name AI crawlers, and for broad rules that block every crawler. Then decide on purpose which ones you allow. Some security or “bot protection” settings can also turn crawlers away, so it’s worth checking those too.

Checking all of this at once

bogl.ai’s free check looks at your homepage. Of the topics in this guide, it covers the page title, description and headings, noindex, HTTPS and robots.txt rules for each AI crawler. Mobile-friendliness is not on its list, so check that one separately. You get a score out of 100 and your top three issues in about 10 seconds, with no sign-up. If you want the fixes, the £39 one-off full report gives them ready to paste, such as robots.txt lines and page titles and descriptions.

Run the free website check