How to get your website into Common Crawl

What the problem is

Common Crawl is a non-profit that crawls the web and publishes the results as a free archive, a new crawl every month or two. Many AI models are trained partly on it. The full report checks whether any of your pages are in the latest crawl.

Why it matters for AI search

If none of your pages are in it, AI models trained on it are less likely to know your business exists from your own website.

You can't submit a site to Common Crawl. Its crawler, CCBot, finds sites by following links from other sites.

How to fix it

1. Make sure CCBot isn't blocked in your robots.txt (see the robots.txt guide).
2. Get links from other websites: local directories, trade bodies, suppliers, local news and community sites. Each one is a route for crawlers to find you.
3. Keep your homepage and main pages reachable without logins, pop-ups that block content, or bot protection that challenges crawlers.
4. Check again after the next crawl is published. It can take a few crawls for a site to appear.

How to check it's fixed

• Common Crawl publishes an index you can search at index.commoncrawl.org. A new full report also checks the latest crawl.