What They Are & How to Fix Them in 2024

Web crawlers (also called spiders or bots) are programs that visit (or “crawl”) pages across the web.

And search engines use crawlers to discover content that they can then index—meaning store in their enormous databases.

These programs discover your content by following links on your site.

But the process doesn’t always go smoothly because of crawl errors.

Before we dive into these errors and how to address them, let’s start with the basics.

What Are Crawl Errors?

Crawl errors occur when search engine crawlers can’t navigate through your webpages the way they normally do (shown below).

When this occurs, search engines like Google can’t fully explore and understand your website’s content or structure.

This is a problem because crawl errors can prevent your pages from being discovered. Which means they can’t be indexed, appear in search results, or drive organic (unpaid) traffic to your site.

Google separates crawl errors into two categories: site errors and URL errors.

Let’s explore both.

Site Errors

Site errors are crawl errors that can impact your whole website.

Server, DNS, and robots.txt errors are the most common.

Server Errors

Server errors (which return a 5xx HTTP status code) happen when the server prevents the page from loading.

Here are the most common server errors:

• Internal server error (500): The server can’t complete the request. But it can also be triggered when more specific errors aren’t available.
• Bad gateway error (502): One server acts as a...