Find & Fix 404 Errors On Your Website
May 04, 2019
How Do You Find & Fix 404 Errors On A Website?
Some of the people who visit your website are going to encounter a 404 not-found error page. Maybe that visitor typed in a URL incorrectly or maybe they clicked on a link that took them to a page that no longer exists on your website.
Whatever the reason they reached a 404 not-found error, you need to know how these visits to error pages are affecting your business.
Why? Because not-found errors create a bad user experience and that bad user experience causes people to leave your website. In short, this means you lose customers.
Can you afford to lose customers? Of course not.
Once found, to stop losing customers due to 404 errors, you need to fix every 404 error that your visitors encounter. There are different ways of fixing errors depending on the nature of the error and the impact the error is having.
How do you find 404 not-found errors? Once found, what options do you have to fix these errors? What tools can help you in this process? Let’s answer these questions in three parts:
- Finding Every 404 Not-Found Error
- Reviewing Broken Link Checkers & Error Monitors
- Fixing 404 Errors On Your Website
Finding Every 404 Not-Found Error
The first step to fixing the error in finding 404 errors is to determine what tools you will use. There are a lot of choices, but before the specific tools, let’s talk about what expectations we should have from those tools, including the overall requirements and the specific types of data these tools should provide.
Requirements for Error Monitoring Tools
Find Errors From All Sources
People can find a 404 error on your website in lots of different places—they can click on a bad link in a search result or find a broken URL in an old email newsletter. Regardless of the error source, you error monitoring tool(s) needs to help you look at every place your visitors might encounter not-found errors. You can’t fix 100% of your website’s 404 errors if your error monitoring tool only is only finding 20% of your website’s 404 errors.
Report On Loss From 404 Errors
Any broken link checker or error monitoring tool you use needs to tell you what the 404 error is costing you. When visitors encounter broken pages, they leave and that means potential customers are clicking away from your website. Or, if Google is encountering lots of errors when crawling your website, that could be costing you rankings in key search results. You need to know how many visitors you are losing due to 404 errors. This helps you prioritize fixing the 404 errors alongside the thousands of other things you have to do.
Sporadic & Continual Nature Of 404 Activity
The error monitoring tool you use also needs to understand that visits to 404 errors are sporadic. One day, nobody will encounter your website’s not-found error page. The next day, one hundred people might find a broken link taking them to a not-found error on your website. Any tool you use needs to constantly monitor your website and alert you when visits to 404 errors increase so that you can fix the problem immediately.
How To Fix 404 Errors
Finding errors is one thing. What you really need though is to fix 404 errors—and not all error monitoring and link check tools will do that. Some tools might tell you where the broken links are located while others might suggest places to redirect the error. The more help the link check tool provides advice on how to fix, the faster you can fix the 404 error and the faster you can stop losing customers due to those 404 errors.
Easy To Install & Configure
Finally, whatever tool you use to check for 404 errors on your site, you want it to be easy to use and easy to install. Time spent configuring a tool is time you could be spending on other tasks.
Data You Need About Your Site’s 404 Errors
What should your error monitoring tools tell you about your website’s 404 errors? What data do you need to make informed decisions about your website’s 404 error activity?
- How Many Visitors Reached A 404 Error? You need to know how many people see a 404 not-found error on your website to help you understand how big a problem 404 errors are for your website and your business.
- Where Did Visitors Find 404 Errors? Most people find your website’s 404 errors from somewhere other than your website. That might be from another website, an email newsletter, a search result, a social network, or a visit from an outdated bookmark. Regardless of where people found the 404 error, you need to know about it.
- What Did People Do After Encountering A 404 Error? Did people stay on your website after encountering a 404 not-found error? Or did they leave (as most people do)? If they stayed on your website, what pages did people look at next?
Reviewing Broken Link Checkers & Error Monitors
All right, now that we understand what to expect and what data we need to find not-found errors, let’s talk about the different types of tools we have available.
How Crawl Tools Work
One of the more popular tools for checking broken links that lead people to 404 errors on your website is the broken link crawl (or scan) tool. Crawl tools, like Screaming Frog or SiteLiner, check every link on your website to determine whether the link leads to a 404 error. (Side note: crawl tools can do a lot more than check for errors.)
These types of tools can be incredibly helpful to quickly evaluate the health of the links within your website. Running a crawl can help you identify which links on your website are broken, which does help you identify the source of the error making it easier to know what to correct. Plus, these tools tend to be easy to install and configure.
Two Problems with Crawl Tools
However, there are two major problems with crawl tools.
- Limited Scope. The biggest problem is that crawl tools only check your website for broken links. What about all the other ways people find broken pages on your website—like broken links contained on social media, email newsletters, offline marketing, and more? As a result, crawl tools miss out on a lot of errors.
- No Sense of Priority. Crawl tools also typically don’t offer any type of priority regarding your website’s 404 error pages. That is because crawl tools do not look at visitors to those pages. Without visit information, crawl tools cannot tell you about the spikes in 404 error activity that are costing you business. Crawl tools are also unable to tell you how many customers you are losing due to 404 errors.
Recommendations On Crawl Tools
A crawl tool is useful and should be used to evaluate the health of the links and pages on your website. However, a crawl tool should never be relied upon exclusively to help you find and fix the 404 not-found errors your visitors encounter.
How Analytics Tools Track Errors
Tracking 404 not-found errors in a web analytics tool, like Google Analytics, is possible and one common method is tracking errors via event tracking. For example, you could trigger the following event onload of the error page:
ga('send', 'event', 'error', '404', location.pathname);
You can also track error pages by using a specific page title on your error page then query page reports in Google Analytics to look at pages matching that title. As you can imagine, this requires a fair amount of customization with event tracking and segmentation.
The good news is that, once figured, analytics tools can give you an accurate picture of how errors are costing you business. For example, you can segment your analytics reports by people who encountered a certain error and see what those people did differently than people who didn’t encounter an error.
Incomplete Help With 404 Errors
There are two pieces of information no analytics tool can provide.
- No Preventative Error Checks. Analytics tools can only tell you what happened—past tense—on your website. You may have 200 broken links that could lead people to a 404 error but if nobody clicks on those broken links, then your analytics tool wouldn’t tell you about them. As a result, only relying on this method puts you in a situation where you are can’t prevent a problem from occurring.
- No Help Fixing 404 Errors. Second, analytics tools don’t provide any help fixing the 404 error. For instance, based on the nature of the 404 error, should you redirect the URL to a working page on your website? Or, can you fix the broken link that is leading people to this error? Relying on this limited information, you will need more time to dig into the error in order to fix the 404 errors on your website.
Recommendation On Analytics Tools
An analytics tool should be configured to help check for errors. The extra data can provide a clearer sense of priority and a clearer sense of what 404 errors cost you. Of course, there are some missing data points so this should be used in conjunction with a crawl tool. As well, using analytics tools to track errors will require some configuration, so if this is something you are interested in configuring, please contact me.
Web Log Analysis
Finding 404 Errors In Your Web Log
Provided you have some means of accessing and reading your web log file, your web log file can provide a wealth of information about the inner-workings of your website.
Among a lot of other data, web log files can provide a list of which URLs returned a 404 error. Like with analytics tools, you can use web logs to get data about how people found the error and, like analytics tools, you can find 404 errors from many different sources.
Another advantage is that web log files look at every user that visited your website. That includes human visitors (your customers) but also automated programs (like robots from search engines). In contrast, analytics tools actively remove bot traffic—for many of the questions analytics reports answer, this makes sense. But, with 404 errors, you want to know how humans and robots are impacted so that you understand the cost to customers and the cost to search engine activity.
Two Problems With Web Log Analysis
- No Soft 404s. If your website returns soft 404 errors, web log files will not work. If your website returns soft 404s, broken pages on your site will look like every other page on your website in your web log (because they will all return a status 200). As a result, you would be unable to find every 404 error on your website.
- No Help Fixing 404s. Web log files are, at best, only going to give you a list of the 404 errors robots and humans encountered on your website. Web log files will not help you fix those 404s. Plus, getting the data you need out of your website’s log file to help you fix the 404 errors can be challenging (and your web log may not even have the data you need). As a result, it will take longer to fix your website’s 404 errors. The longer it takes to fix, the more business you will lose due to 404 errors.
Recommendations On Web Log Analysis
Web logs can be incredibly useful to understand how your server is handling your visitors if you have a tool to help you parse the log file. As a result, this tends to only make sense for larger websites where the impact of errors will be worth the greater amount of work to find and fix those errors.
External Link (or Backlink) Tools
What Are Backlink Tools?
One way to find 404 errors on your website is to see if any other websites are linking to broken pages on your site. When other websites link to you, that link is referred to as an external link (as in, external to your website) or a backlink (as in, a link linking back to your website).
For example, a tool like Moz’s Link Explorer can help you identify what websites are linking to your website. You can then use that data to find which websites are linking to 404 errors on your website. (To be clear, Moz’s Link Explorer and other backlink tools can help you find a lot more than broken links as well.)
Helpful for SEO
Looking for broken links that lead to 404 errors on your site can be especially helpful for search marketing purposes. Within search marketing, links from external websites help improve your rankings within search results. A link from an external website that takes people to a 404 error on your website, won’t help search marketing efforts. But by fixing that error you can reclaim that backlink and reclaim the benefit the backlink provided.
Helpful, But Still Not A Complete Picture
At best, external link tools will never provide a complete picture of the 404 errors your visitors encounter. Because these tools do more than look within your website for broken links, external link tools offer more information than link crawlers discussed above. So, with a backlink analysis you will find more 404 errors than you would when using a crawl tool.
However, there are still lots of other ways people can find 404 errors on your website that are missed by external link tools. For instance, people can find broken links leading to 404s on your website on social networks, in email newsletters, from old bookmarks, and more. Backlink tool can’t find 404 errors resulting from those sources.
This is an important tool to use, but one to use on a less regular basis. Instead, I’d recommend looking for these errors as part of quarterly tech SEO checks.
Google Search Console
What is Google Search Console?
Google Search Console is a free tool provided by Google and provides detailed information about how Google sees your website. This includes what errors Google sees. Once in Google Search Console, under Coverage, select Excluded, and then scroll down to see if there are any not-found pages listed. If so, click this to see the full list of errors Googlebot has found related to your site.
Google Search Console is a fundamental tool and provides a wealth of data. The error report is certainly no exception. With this report, you can find a wide range of errors that other tools were unable to surface simply because those tools can’t search the web on the same scale as Google.
Helpful, But Still Not A Complete Picture
Of course, Google Search Console’s error report—like all the tools discussed in this post—is far from a complete picture of your website’s errors. For all of Google’s crawling power, they can and do miss broken links that lead to errors. As well, like with external crawl tools, errors can exist in places away from Google’s purview—such as in email newsletters, social media sites, or offline marketing.
If you haven’t already, you need to setup Google Search Console for your website and you need to look at the reports in Google Search Console on a regular basis. This includes looking at and using the Coverage report. But, again, don’t use this tool in isolation. Instead, use Google Search Console alongside the other tools discussed for the best results.
How To Fix 404 Errors On Your Website
You Found It, Now Fix It
Now that we’ve found the 404 not-found errors, what do we do? There are four methods you can use to fix 404 not-found errors on your website.
- Redirect the 404 error somewhere else. With a redirect, you route people from the error page to a working page on your website. This prevents people from reaching the error. For example, if people are reaching an error page with the URL /specials, you can tell your server to redirect people to the working page /special instead. Just make sure the page you are redirecting to is relevant. For example, you wouldn’t want to redirect an error URL of /specials to /contact because that would leave users confused why they are on a contact page when they were expecting to arrive on the /specials page. Learn more about redirects.
- Correct the source link. If the broken link sending traffic to the not-found error page exists on your website, and is under your control, you can correct the broken link at the source. This is also true for not-found errors on other properties within your control, such as social networks. However, most broken links are located on sources you don’t control so in those cases you can’t fix the broken link and can only redirect people to the correct page.
- Restore deleted pages. Sometimes, you delete pages from your website and people still come to your website to find those pages you deleted. If somebody tries to find a deleted page, they’ll get a not-found error. That might be the point and there may be a good reason for the removal (see fix #4 below). However, if there is still a lot of demand for removed content, you may want to restore the deleted page.
- Ignore the not-found error. Sometimes, for low priority not-found errors that aren’t costing you customers and aren’t affecting your SEO performance, you can simply ignore the 404. In other cases, it is a page you removed purposefully and there is nowhere relevant to redirect the error URL to (for example, you removed a product from your website and there is no new version of that product to redirect to). It may not feel right to leave an error in place, but sometimes fixing errors just isn’t the right solution. By leaving the 404 error in place, that broken page will fall out of the search index and stop getting traffic (eventually).
Running a website for your business requires a lot of work. You must invest a lot of time, money, and effort to get people to notice, visit, and use your website. The last thing you want is to lose visitors as a result of 404 errors. You need to find and fix every 404 error that is costing you business. This requires using a multitude of tools to get a complete picture of the errors on your website and what those errors are costing your business. As well, the multitude of tools needs to be used on a regular basis since 404s can resurface in sporadic intervals. If you need help, please contact me today and let’s talk about what errors (404 or otherwise) are holding you back.