Has Google indexed all the pages on your website? Indexing means that Google not only crawled the pages on your website, but they processed those pages and they moved those pages into their database. And then once they’re in their database or in their index, those pages can be retrieved for search rankings, for AI citations, and so on. So it’s very important that your page end up in the index. You want your pages to be indexed without any issues. So one way that you can tell whether or not your pages are indexed is in Google Search Console’s page indexing report. So the page indexing report in Google Search Console can be accessed by clicking pages in the sidebar.
When you go here on the page indexing report, the first thing you’re going to see is a top line number of how many pages are not indexed and how many pages are indexed. Now the important thing to know is that these two numbers are fine, whatever they are. Plenty of sites have lots and lots of pages that are not indexed. That’s okay.
If you have pages that are not indexed, equally a lot of sites have way more pages not indexed than they do index, and that’s also okay. There’s no magic ratio between these two numbers. There’s no number that this should be. What you want to make sure of is that the right pages are not indexed and that the right pages are indexed.
The other thing that you can look at in this top section is a trend line of how things are going. I tend to find it’s a little helpful if you click off on these to see how each individually is trending. It’s especially true if you have a bigger difference between how many pages are indexed versus not indexed. So in this case the indexed pages are pretty stable over time, maybe a little bit of fluctuation here and there, but overall no major fluctuations.
If you’ve recently added a bunch of articles or pages to your website and you see indexing jump up, then that makes sense. If you see indexing jump up and you didn’t add a whole bunch of pages to your site, then you need to dig in and see what’s going on. Going over to the not indexed side, we can see how that’s trending and we can see that this is generally trending down. Here again, let me just emphasize that there’s no magic direction that these need to go in. There’s nothing good about not index pages going down. There’s nothing good about index pages going up. There can be good or bad reasons for these trending in either direction.
So, you really need to dig in. You need to understand the details about what is not indexed and what is indexed and whether or not there’s a problem. So, the first thing that you can do is you can view data about indexed pages. If you click on this link, you’ll be taken to a list of example URLs that are indexed. They’ll show you up to 1,000 URLs that are indexed. In this case, there’s only 144 pages that are indexed, so I’m going to see all of them here. But, for sites that are larger, you’re only going to get up to 1,000 of these.
So, you can go through and you can review all these URLs and make sure that they are actually URLs that should be in the index. Sometimes we go through this when we’re doing audits for clients and we start to see that, oh wait, there’s URLs in the index that are listed here that actually shouldn’t be. That are are actually problematic URLs or URLs that should not exist on this website or URLs that are not the canonical URL, right? Things that we don’t actually want on here.
So, in that case, we need to get rid of those URLs and move those out of the index status because we don’t want those pages in the index. So, you want to go through and review this and and see what’s going on. The more important thing to dig into though is this table, why aren’t pages indexed on your website? And so, there’s lots of different reasons why Google may not index something. The important thing to know is that some of these reasons that pages are not indexed are perfectly fine reasons. These are just normal things that happen on a website and you would expect these to happen and they’re not going to affect your site’s performance or hurt your site in any way. It’s totally okay if you have pages that are not indexed because they are 404 or because they redirect or because they have a no index tag.
In those cases, these are valid things, right? If you recently removed a bunch of pages from your website, you’re going to have 404s. Google is recognizing that you deleted that page and that deleted page now returns a not found 404 error. It should return a not found 404 error because you removed and deleted that page. Same goes with a redirect. You recently redirected one URL to another URL. Google maybe found the URL that redirects and they also followed that redirect and went to the other page.
So, the page with redirect, that’s telling you that these URLs are ones that Google found that yes, you added a redirect. We are not going to put the redirected URL in the index, but what you want to then go check is if the place the redirect goes to, the redirect destination, that that is in the index. So, it’s totally fine if you have some of these pages in here for some of these reasons. What you want to watch out for are the reasons that might indicate a problem.
So, if you have things like soft 404s coming up, that’s not something that’s valid, right? You don’t want things to be a soft 404. Soft 404 meaning that your server says that the page is working properly, but the content on the page says that it’s an error. That’s not something that’s valid, so that’s something that you would want to find and fix if it was in fact reported here. Equally, if you have any issues with duplication that might be coming up, that’s usually a problem and something that you want to figure out why that’s coming up and how to fix that.
So, you need to go through and separate these into categories that represent a valid state and things that represent a potential problem. Couple of the categories that are right in the middle of that spectrum of is it a problem? Is it not a problem? Are crawl currently not indexed and discovered currently not indexed.
So, crawled currently not indexed is basically Google’s catch-all category. So, this is the category where it is problematic in some way. There’s some kind of issue with the page more than likely that’s going to prevent it from ending up in the index, but Google isn’t too sure what that category is or what that reason is or at least it’s not one that they’re going to tell us about here, and so they throw it in crawled currently not indexed. And when you have pages in crawled currently not indexed, what you need to do is you need to click into crawled currently not indexed, and you need to go through and you need to figure out, well, why are these pages listed here? What is it about the page that potentially is causing this to not be moved into the index? Because crawled currently not indexed means, “Hey, Google found the page. They saw the page.
They looked at it, right? They crawled it, but they decided that there’s something wrong with it.” So, as you go through and look at the pages, you can start to see, well, why is that? In some cases, it’s simply that the page is new, and so it’s just going to take a little while for that page to move in. In other cases, there might be a technical issue with the page.
So, Google tried to crawl the page, but then they saw that the page was blank because there was a problem with JavaScript or there was a problem with CSS, and so as a result of those problems, they weren’t able to successfully crawl the page, and that prevented them from being able to put the page in the index. In other cases, it’s not a technical issue, but a content issue, and so something about the content is preventing this page from moving into the index. Could be that the content is outdated. Could be that the content is considered unhelpful. Could be considered just low-quality content.
So, in a lot of cases, we see pagination coming in here. Those aren’t pages Google maybe always wants to rank in search results. Those are okay. Sometimes it might be older blog posts or older articles that you see listed in here. That could be okay and an indication that, yeah, maybe you should actually update this. You should maybe dig in and and understand what’s going on with the page and refresh it and make it more helpful, make it more high-quality content on here.
The other category to take a look at is discovered currently not indexed. So, discovered currently not indexed means that Google knows about the page, but they haven’t looked at it. Really, what this category should be called is discovered currently not crawled or indexed. So, basically, Google knows that the page exists, but they haven’t bothered to go back and even look at this page. I see a lot of pages end up here that are simply new. These are just new pages that were recently added to the site and so Google is just going to take a little bit of time to go back through.
You can also see pages in here that are orphans, pages that don’t have many internal links pointing to them. So, Google just doesn’t have much of an incentive to go and crawl those pages. Sometimes this can also point to a content quality issue where Google maybe thinks that this page isn’t going to contain very high-quality content and so that’s something that’s going to prevent them from going back and crawling or indexing this page. For any of these URLs and in any of these page indexing categories, you can dig into more details within this.
So, for example, let’s take this article here, this blog post here. And what I can do if I want to inspect this a little bit more is I can click the magnifying glass icon, inspect URL. And this will retrieve data about this page from Google and we can see if there’s any more details that we have. First thing to double-check is that the page is still not indexed. Sometimes you have a page that’s listed on the pages report that says, yep, it’s not indexed, and then you click over here and you realize, oh wait, it actually is indexed. And so it was just a misreport on the page indexing report. And that’s okay.
But other times you click in and you realize, “No, that’s actually valid that the page is not indexed.” And that’s what we see here. So, in this case, we can see that they found the page in the site map, but nothing else. And because this isn’t discovered, currently not indexed, Google hasn’t crawled the page, and we don’t know anything about the page’s status. Could they actually fetch the page? Could they discover the page? Could they crawl the page? Could they index the page?
Right, we don’t know any of these things. But, if Google has actually crawled the page, then you will have some of these details here, and that can be helpful to debug and look at things. Now, what we can do to rule out some of the technical issues that may prevent Google from being able to successfully crawl this page is test the live URL. So, if I click test live URL, this is going to do a live hit of the website and try to fetch that page’s content. And we can tell whether or not there’s an issue with the page or not because if there’s an issue with the page, then Google won’t be able to crawl it, they won’t be able to actually go there and retrieve anything from the page. In this case, it comes back with green check marks. The page is available to Google, and it says that the page can be indexed.
One thing that’s helpful is we can view the tested page. So, when I click view tested page, I can look at all the HTML code that comes back. Unless you’re really good at reading HTML, this is going to be a little tricky to work with. So, what you can do here is you can hit copy, and then you can bring this into Notepad or something else, and you can search through and you can see if all of your content is there, you see everything present on the page or not. Sometimes that’s a little easier to do for a deeper dive debug. In other cases, if you’re just going for something a little lighter, you can click on screenshot, and you can see what the page looks like. This isn’t going to give you a whole screenshot, it’s going to get cut off after a certain point, but it will at least give you an idea as to whether or not the page appears to have loaded correctly.
In this case, it did appear to load correctly, but in other cases, I’ve run live tests and you see that oh, the design is really broken because of an issue with JavaScript or an issue with CSS. And that’s going to prevent the page from loading correctly, and that might prevent Google from wanting to index the page. You can also click on more info to get some more details about the page. In particular, I like paying attention to page resources. This tells you if there’s anything that could not be loaded.
So, any issues with reading the JavaScript files on this page. You’re always going to have a few pages that or a few resources that can’t be loaded, but they’re usually going to be things like tracking code, which is what we see here. Gstatic, DoubleClick, right? These are just kind of unnecessary files to load the page, right? These are kind behind the scenes operational kind of things on here.
So, we don’t need to worry if they can’t load those. But in other cases, I look at this and it’s actually core JavaScript files, and that is going to be a problem and that is going to prevent Google from being able to successfully load the page. So, it’s a good way to debug things and look at what’s going on. Assuming you’ve gone through and you’ve checked all the technical things, not just live test, but really dug in and looked at the technical aspects of the page and looked at the content on the page, see if there’s anything else wrong. Assuming everything checks out and you think, “Okay, I’ve improved the page. I fixed the problems.” You can click request indexing. And when you click request indexing, this will submit a request to Google that suggests that they go back and they recrawl this page.
They put it in their crawl queue. And you can, in many cases, see that requesting indexing might help move that page into the index. It doesn’t always help. It’s not a guarantee that that page is going to move in, but it can help.
And if you actually dug in and you fixed the problem on this page, you updated the content, you added internal links to the page, you fixed technical problems that exist, and so on, it can be successful to get it to move back into the index. Now, along with Google Search Console, we can also check whether or not Bing has indexed any pages on our site. So, in Bing, we have a report called Site Explorer. So, in the sidebar, click on Site Explorer, and then it brings up this page.
Now, if you expand this, you can see a hierarchy of all the URLs on your website, all the sections on your website. So, for example, I can see all the different articles I have, but then I can also see the blog, and I can look at my blog, and I can see whether or not everything in my blog is indexed or not. And so, in the case of the blog, I can see that 86 pages are indexed. There are no errors, there are no warnings, there is nothing excluded. That is what I want to see. But, if I go back up to the overall, I can see that there are 24 warnings and four excluded pages. And so, I want to understand, well, why is that?
What is actually getting excluded? What is actually getting kicked out of the index? So, to help me dig in and explore that, I can go up here to the filter, and I can start going through and look at the different categories of issues. So, for example, I can look and see if there are any URLs with crawling issues. Are there dead links on here? And in fact, there are some that are coming back as 404s and 410s, and I can dig in and understand, well, why are those 404s? And why maybe I are those happening?
Are those valid 404s? Are these pages I deleted, or are these pages that I intended Bing to actually move into the index. And same goes for all the other categories. I can look at URLs with other issues, or URLs that were selected, that were not yet crawled. Or if there is any issues with pages being blocked from crawling, or pages that are not indexed, or URLs that are redirecting, and so on. So, it’s a good way to start digging in and understanding what issues may be present on my website, and what kinds of URLs I might need to focus on and address.
If you have any questions about working with either of these reports in Google Search Console or Bing Webmaster Tools, or generally just need help figuring out why pages on your website aren’t indexed or other technical SEO issues, please let me know. You can email me at [email protected]. If you liked this video, please subscribe. Thank you for watching.