What is “Indexed, though Blocked by robots.txt” in Google Search Console? Why does it happen? Is it a problem? What do we do about it? Let’s dig in…
One of the most misunderstood parts of using robots.txt is the idea that a Disallow automatically prevents a page from ranking. This Disallow does not do this. A Disallow rule only controls crawling, not indexing.
This is because crawling and indexing are independent behaviors for Google. So, a page can still be indexed (and rank!) despite being blocked from crawling.
This seems wrong. Why does Google index pages it can’t crawl?
Google relies on a variety of signals to decide whether something belongs in the index. Crawling is just one input. Others include internal links, external links, sitemaps, general site structure and more. So, if you have a page with strong internal linking, Google may decide it should be included in the index based on those signals alone, even if it is blocked by robots.txt and Google cannot crawl it.
You can see pages that have been indexed despite the disallow in Google Search Console’s Page Indexing report. This report will show a warning of “Indexed, though blocked by robots.txt.”
This isn’t always a problem. There are cases where this state is perfectly fine. In some cases, it might even be the desirable or intended outcome.
A good example is PDF files. You may not want Google to crawl them because they are large and crawling is expensive for your server. However, if the PDF ends up indexed and occasionally ranking, that may be acceptable.
These pages typically do not rank particularly well, but if discoverability is fine and crawl budget is a concern, leaving them indexed but blocked with a Disallow is a reasonable choice to control crawl budget.
Sometimes, though, “Indexed, though blocked by robots.txt” creates problems.
As an example, think of filter or faceted URLs. If these URLs are blocked from crawling but still get indexed, they can start competing with the unfiltered, root version of the page. Google may treat the blocked URL as closely related to the real one and begin surfacing it in search results. That can affect the performance of the root version of the page.
In that case, blocking the crawl is the wrong control. The fix is to switch to controlling indexing instead of crawling.
If you need Google to not index the page at all, you need to switch from robots.txt Disallow to a meta robots tag on the page itself:
Use:
<meta name="robots" content="noindex">This explicitly tells Google not to index the page.
Here’s the catch: Google must be able to crawl the page to see that noindex tag. If the page is blocked via robots.txt, Google can’t see the HTML tags (including the meta robots tag), and therefore can’t see the noindex directive.
The big takeaway here, though, is that you cannot control both indexing and crawling at the same time. You have to choose:
- Control crawling (using robots.txt)
- Control indexing (using meta robots)