Noindex vs. Nofollow vs. Disallow: When to Use Each One

There are a few different ways to control how bots interact with your website. The three that come up most often are:

• meta noindex
• meta nofollow
• rel nofollow
• robots.txt disallow

At first, these seem like they should all do roughly the same thing: keep bots or search engines away from certain pages. However, they each do very different things and control different behaviors. In tech SEO audits, we commonly see these being used incorrectly, negatively affecting SEO performance.

Using the wrong type of control can create indexing problems, block bots from seeing content they need to see, or accidentally leave pages visible in search results that you thought were hidden.

To understand which control to use, the better question is to understand what you are actually trying to accomplish.

• Are you trying to keep a page out of Google’s search results?
• Are you trying to stop a page from being included in AI training data?
• Are you trying to stop bots from crawling a URL?
• Are you trying to explain the nature of an outbound link?
• Are you trying to protect private or internal content?

Each of those goals requires a different solution. Let’s walk through the differences.

Crawling and Indexing

Before we get into each directive, we need to separate two concepts: crawling and indexing.

Crawling is when a bot discovers and fetches a URL. Bots discover URLs from internal links, XML sitemaps, backlinks, and other sources. Once a bot finds a URL, it may request that URL and fetch the page’s content, images, scripts, styles, headers, and other information.

Indexing happens after crawling. This is where a search engine evaluates the content and decides whether that page should be eligible to appear in search results.

That distinction is the key to understanding exactly how and when to use a noindex, disallow, or nofollow. Keep in mind that:

• A page can be crawlable but not indexable.
• A page can be blocked from crawling but still discovered.
• A page can even appear in search results even though it is blocked in robots.txt.

Here is the simplest way to think about these controls:

• Disallow tells bots not to crawl a URL.
• Noindex tells bots not to index a page or rank it in search results.
• Meta robots nofollow tells bots not to follow links on a page.
• Rel qualifiers, like nofollow, sponsored, and ugc, explain the relationship between your page and a specific link.

Disallow: Controlling Crawling

The disallow is placed on a robots.txt file. The robots.txt file is a plain text file placed in the root directory of a website. It provides instructions for robots, including the disallow instruction, telling robots which directories you would prefer they not crawl.

There are three standard fields we can use in a robots.txt file:

• user-agent: The name of the bot we are applying the rules to.
• allow: A directive or rule that the page or file referenced should be crawled. (This is the assumed, default behavior.)
• disallow: A directive or rule that the page or file referenced should not be crawled.

This code tells all bots that respect robots.txt not to crawl URLs in the /cart/ directory:

User-agent: *
Disallow: /cart/

This is what Amazon’s robots.txt file looks like:

How Robots.txt Rules Are Applied

You can define Disallow or Allow rules for specific bots. However, specific rules supersede generic rules. Take this example:

User-agent: *
Disallow: /admin/
User-agent: Bingbot
Disallow: /specials/

You might think that in this example Bing would not crawl the admin directory or the specials directory. After all, the “User-agent: *” says that those rules apply to all bots. However, Bing (and Google and others) will ignore the generic, global section if they find a specific set of instructions for their bot.

That means you would actually need to write the robots.txt file this way to block Bing from crawling both the admin and specials directories:

User-agent: *
Disallow: /admin/
User-agent: Bingbot
Disallow: /specials/
Disallow: /admin/

It is also important to remember that robots.txt rules are case-sensitive. In the previous example, there is crawl block for the /admin/ directory. However, if Google attempts to crawl /ADMIN/ or /Admin/, that rule would not apply and Google would attempt to crawl. It is easy in certain environments to let URLs with different casing sneak out. Ideally, your website forces casing with redirects. If not, check if you need to adjust your log files.

Disallow Does Not Control Indexing

The most important thing to remember is that the disallow statement controls crawling. It does not control indexing, so using a disallow does not guarantee that the URL will stay out of search results.

In some cases, that blocked URL may still show up in search results. Usually, Google will not have much information about the page because it was blocked from crawling it. Even if Google cannot see the page itself, the URL can still be discovered. Any URL that is discovered can be indexed. This is why robots.txt is not the right tool if your primary goal is “I do not want this page ranking.”

Google may still discover a disallowed URL if it is linked from your website or from another website. Google might not be able to crawl the page, but it can still know the URL exists and any URL that Google knows about can be included in search results.

I see disallowed URLs ending up in search results more often when the URL is heavily linked from the website. So if you truly do not want bots discovering those URLs, you may also need to rethink how those URLs are linked internally in addition to the robots.txt disallow.

For example, a cart page on an ecommerce site may not need to be a crawlable link. Users need to access the cart, but bots generally do not. In that type of case, you may use a button or JavaScript-triggered action instead of a standard crawlable <a href=””> link.

That is not something you would want to do for important content pages. For utility pages like cart, checkout, account, or login pages, it can be the right answer to limit the discoverability of disallowed URLs. The more you can limit discoverability, the better the chances the page will not be indexed even though it is blocked by the robots.txt file.

When Disallow Makes Sense

A robots.txt disallow can make sense when the goal is to stop bots from crawling a URL, file, or section of the site.

For example, you might use disallow for filtered or faceted URLs that allow for infinite amounts of filter combinations. If the various filter combinations create unique URLs, that can result in thousands or millions of additional URLs, none of which are useful for bots to crawl.

You may also want to use the disallow for files or directories that create server strain if they are crawled too aggressively by bots. Be careful, though, not to block files that are necessary to understand a page. I have seen clients disallow CSS files to save their servers, but bots need those CSS files to load and understand the HTML pages.

A disallow also helps manage what sections of the website AI training bots can and cannot access. If there is content that you do not want included in AI training material, a disallow will prevent AI bots from accessing that content (at least for the AI bots that support the robots.txt file). For example, CNN has a Disallow for all generative AI training bots.

Noindex: Controlling Indexing

A noindex directive tells bots not to include a page in search results. The most common way to add this is with a meta robots tag in the page’s <head>:

<meta name="robots" content="noindex">

This means bots can still crawl the page and can still see the content. Bots can also still follow (crawl) links that are contained on the page. What this directive says, though, is that the page itself should not be included in search results.

That makes noindex the better answer when the issue is indexation.

For example, you might use noindex for:

• Thin pages that are useful to some users but should not be search landing pages
• Ad landing pages
• Internal search result pages
• Some tag or archive pages
• Filtered pages that should not rank
• Duplicate or near-duplicate pages that need to remain accessible

The key is that Google needs to crawl the page to see the noindex. That means you usually should not block a page in robots.txt if your goal is for Google to see the noindex.

If you only want to prevent Google from indexing, but let Bing or other search engines continue to index the content, you can alter the tag name from “robots” to specify the name of the bot (“googlebot” or “bingbot”). For example, this tag would prevent Bing from indexing while letting Google continue to index the page:

<meta name="bingbot" content="noindex">

Noindex Is Not a Content Quality Strategy

Noindex can be useful for low-value, unhelpful pages, but it is not a substitute for fixing content.

If you have old blog posts, outdated resources, thin articles, or inaccurate pages, the first question should not be about a noindex tag. Instead, the first question should be whether this page should exist on your website. The better answer is usually to improve, consolidate, redirect, or remove the content:

• If the topic is still relevant, update the page. 
• If there is a better page that satisfies the same intent, redirect the low-quality page to the better page. 
• If the page is no longer useful and has no good replacement, remove it.

Noindex can make sense when the page still has a purpose for users but should not be a search landing page. But if the page is outdated, inaccurate, or otherwise low quality, noindex is often the wrong tool to use or, at best, a short-term measure.

What About PDF Files and Other Non-HTML Assets?

Meta robots tags work inside HTML. That means they do not work the same way for PDFs, images, and other non-HTML files. For those files, you can use an X-Robots-Tag HTTP header.

X-Robots-Tag: noindex

This tells bots not to index the file. This can be useful for PDFs, images, documents, or other files that you do not want appearing in search results.

Just like with a meta robots noindex, bots need to be able to access the file to see the header. So if the file is blocked in robots.txt, Google may not be able to see the X-Robots-Tag.

Common Questions: Noindex vs. Disallow

How Do Disallow + Noindex Work Together?

One of the most common mistakes I see working with clients is that they have added both a robots.txt disallow and a meta robots noindex to the same page.

That can sound logical at first:

“We don’t want Google crawling this page, and we don’t want it indexed either.”

The problem is that these two controls conflict with each other. If robots.txt blocks Google from crawling the page, Google may never see the noindex tag. If the bot does not see the noindex, then the page will still be in the index. When it ranks in search results, there will be a message saying that “No information is available for this page”.

To understand this, it helps to think through the three ways these commands can be combined and how these combinations affect indexing and crawling.

In Scenario 1, the page with a noindex setting will not be included in a search result. However, a robot may still crawl the page, meaning the robots can access content on the page and follow links on the page.

In Scenario 2, the page will not be crawled but may be indexed and appear in search results. Because the robot did not crawl the page, the robot has no information or content from the page.

Scenario 3 will operate exactly like Scenario 2. This is because when a Disallow is specified in the robots.txt file, a robot will not crawl the page. If the robot doesn’t crawl the page, it will not see the meta tag indicating not to index a page.

How to Fix Indexed, Blocked by Robots.txt

If a page is already indexed and blocked in robots.txt, you generally have two options.

The first option is the cleaner long-term path:

1. Remove the robots.txt disallow.
2. Add a meta robots noindex to the page.
3. Let Google crawl the page and process the noindex.
4. Remove or reduce internal links pointing to that page if they are not needed.
5. After the page drops from the index, decide whether to re-add the disallow.

The second option is more of a temporary suppression path:

1. Keep the robots.txt disallow in place.
2. Use Google Search Console’s Removals tool to temporarily hide the URL from search results.
3. Re-submit the removal request as needed if the URL returns after the temporary removal expires. (Google says it expires after 6 months, but at Elementive, we have seen it expire much quicker than that.)

The first option is usually better if you want a permanent fix. The second option can make sense when you absolutely do not want to open the URL back up for crawling, even temporarily.

Should You Noindex About Internal Search and Filter Pages?

Internal search pages, faceted navigation pages, and filtered listing pages are some of the most common places where this gets complicated.

For example, an ecommerce site might have URLs for every possible combination of:

• Category
• Brand
• Size
• Color
• Price
• Rating
• Availability
• Sort order

A directory site might have the same issue with filters for:

• Location
• Specialty
• Service type
• Features
• Availability
• Price range

Some of those pages may be useful search landing pages but many of them will not be.

Do not apply a blanket rule. I have seen too many sites get into problems with blanket rules that cover all page types. This situation requires careful analysis and nuanced decisions. 

The first step is to decide which pages are actually valuable. This requires asking:

• Are people searching for this type of page?
• Are people asking AI tools prompts related to this type of page?
• Would this page be a good entry point from search results?
• Is this meaningfully different from another page?
• Does this page satisfy a real user need?
• Is this page already covered by a better category, subcategory, or landing page?
• Is this page useful for AI systems to understand the business, products, or services?

If the page has search value, meaning that people would want to find this in search results, then you want to keep it crawlable and indexable.

If the page is useful to users but should not rank in search results, use noindex. This is often the case for deeper filter pages.

For AI crawlers, you may need a separate rule set. A page that is not useful for Google’s index might still contain useful information for AI systems. For example, a noindexed page describing a slightly different product use case might still be useful for AI training or retrieval. But thousands of thin faceted URLs with no meaningful unique content probably are not.

You have to decide how each bot and system should work with the page and apply the disallow or noindex controls accordingly. That means you may end up with a more nuanced setup.

For example:

User-agent: Googlebot
Allow: /search/
User-agent: GPTBot
Disallow: /search/

Then, on the search result pages themselves, you might include:

<meta name="robots" content="noindex, follow">

In that setup, Googlebot can crawl the page, see the noindex, and keep the page out of Google’s search results. Meanwhile, GPTBot (OpenAI’s training bot)  is blocked from crawling those search result pages, and those pages will not be included in training data.

That is a more specific strategy than simply blocking everything or noindexing everything. You would need to extend this consideration to each AI bot across each page type on your website.

Noindex or Disallow Cart, Checkout, Login, and Account Pages?

Utility pages are another area where clients often ask whether to use noindex or disallow.

This all comes back to the overall goal. The answer for one client is never the same as the answer for another client.

If the goal is simply that you do not want these pages ranking in search results, use noindex.

If the goal is that you do not want bots crawling these pages at all, use robots.txt disallow.

If the goal is to protect server resources, reduce bot abuse, or prevent access, robots.txt and noindex are not enough. In that case, you want to restrict access in your server configuration. For example, you might use Cloudflare, server rules, authentication, or other access controls to prevent bots from hitting sensitive or resource-intensive URLs.

Other clients I’ve worked with don’t care if bots crawl or if these pages rank in search results. Since there isn’t a strong preference, the right answer is to not use a disallow or a noindex.

Blocking Access to Staging, Preview, QA, and Internal Environments

Staging, preview, QA, and internal environments should not rely on robots.txt or noindex.

Those environments should be protected at the access level. That usually means one or more of the following:

• Password protection
• VPN access
• IP allowlisting
• Server-level blocking
• CDN-level blocking

A robots.txt disallow can still allow URLs to be discovered. A noindex still allows bots to crawl the content. For private or internal environments, that is not strong enough protection.

If a staging or QA URL has already been indexed, lock down access first. Then use Google Search Console’s Removals tool to temporarily remove the URL from search results.

I have a separate article on staging environments and SEO, so I won’t go deeper into that here. The key point, though, is that private environments need access control, not a disallow or noindex.

Pagination: Usually Keep It Crawlable and Indexable

Pagination is another place where noindex and canonical tags are often misused. You usually want each paginated page to be crawlable and indexable. Each page in the sequence contains different items, posts, products, or listings. Because of that, page 2 is not the same as page 1.

In most cases:

• Page 2 should set the canonical to page 2. Page 3 should set the canonical to page 3. And so on.
• Paginated pages should not canonicalize back to page 1.
• Paginated pages should not be set to noindex.
• Paginated pages should not be blocked with robots.txt disallow.
• Bots should be able to follow links through the paginated sequence.

Canonical tags are a separate topic, so I won’t go too deep on that here. The main point is that pagination is usually not the same as duplicate content. If you want more detail, see my article on canonical tags.

Rel Nofollow and Other Link Qualifiers

The other type of nofollow is rel=”nofollow”. This is applied to a specific link:

<a href="https://www.someotherwebsite.com/" rel="nofollow">Link</a>

This does not mean “do not crawl this link” in the same way a robots.txt disallow controls crawling. Instead, it qualifies the relationship between your page and the page you are linking to.

This is where a lot of people ask about the idea of “leaking SEO value.” That is not the right way to think about it.

Rel qualifiers are not really about managing SEO value. They are about explaining the nature of the link.

Google supports several link qualifiers:

• rel=”nofollow”
• rel=”sponsored”
• rel=”ugc”

For example, this link has a rel sponsored attribute added:

<a href="https://www.somepartnerwebsite.com/" rel="sponsored">Link</a>

Use sponsored for paid, advertising, affiliate, or partnership-driven links.

Use ugc for links added by users, such as links in comments, forums, reviews, profile bios, or other user-generated content.

Use nofollow when you do not want to imply endorsement or when the link does not fit cleanly into another category.

For normal editorial links, you usually do not need a qualifier.

If you are citing an authoritative source, linking to a useful reference, or pointing users toward a trusted resource, that link can usually be a normal followed link. That is part of how the web works.

In fact, linking to useful, authoritative sources can support trust. If every outbound link on your site is nofollowed, sponsored, or otherwise qualified, that can start to communicate something strange about your editorial standards.

One other thing to remember is that you should never use qualifiers on internal links. You do not want to or need to qualify links from one page of your site to another page on your website. Internal links within your website are perfectly normal and expected by bots.

User-Generated Content Links

User-generated content needs special attention. If users can add links in comments, reviews, forums, listings, bios, or vendor descriptions, those links should usually include rel=”ugc”.

For example:

<a href="https://www.example.com" rel="ugc">User-submitted link</a>

You may also combine qualifiers:

<a href="https://www.example.com" rel="ugc nofollow">User-submitted link</a>

The biggest thing to keep in mind is that rel=”ugc” is not a moderation system.

It does not protect your users from clicking a bad link. It does not stop spam from appearing on your site. It does not make an unsafe link safe.

You still need moderation rules, spam detection, URL validation, and policies around what users are allowed to submit.

In some cases, the right answer is to remove the link entirely or outright prevent users from adding links in that area of the site.

Meta Robots Nofollow: Usually Not Needed

There are two different kinds of nofollow. The first is meta robots nofollow. This is applied at the page level in the <head>:

<meta name="robots" content="nofollow">

This tells bots not to follow, or crawl, any links found on this page. In most cases, this is not what you want.

For example, let’s say you have a blog tag page. You may not want the tag page itself indexed, but you probably do want Google to follow the links from that tag page to the blog posts. This is a way for bots to find links to those blog posts.

In that case, this would make sense:

<meta name="robots" content="noindex, follow">

Or, since follow is the default behavior, you can simply add this tag to let bots crawl links found on the page but keep the page itself out of search results:

<meta name="robots" content="noindex">

There are some situations where a meta nofollow might be appropriate, but they are rare. Most of the time, if a page is available for bots to crawl, you still want bots to follow the links on that page.

Testing Noindex, Nofollow, and Disallow

After you add any of these directives, you need to test them. You need to confirm the noindex, disallow or nofollow is present and test if bots are seeing it, respecting it, and behaving the way you expected. There are a few different tools that you can use to review how these directives are used on the site. 

Google Search Console: URL Inspection

For Google, the easiest place to start is Google Search Console’s URL Inspection tool.

With URL Inspection, you can check whether:

• Google can crawl the URL
• Indexing is allowed
• A noindex tag is present
• The page is blocked by robots.txt

In Google Search Console, enter a URL in the bar at the top of the screen.

After the results load, within the coverage report, you can see if crawling and indexing are allowed. In this example, both are allowed—which is the intended response. If, however, I had specified a noindex or disallow for this page, the crawl or indexed allowed answers should be a “no”.

Learn more about URL Inspection in Google Search Console:

Robots.txt Testing Tools

For robots.txt rules, there are a few different robots.txt testing tools available that will confirm that the right bots are allowed or blocked.

This is especially important when you have bot-specific rules. For example, you may allow Googlebot but disallow GPTBot, or allow one directory for Googlebot but block another directory for other bots.

TechnicalSEO.com robots.txt testing tool and Rank Math’s Robots.txt tester both provide a great, free tool you can use to test your current robots.txt file. With both of these tools, you can either test your live robots.txt file or use the editor to add new robots.txt rules to evaluate impact. You can also change the User Agent, to test if some robots can crawl (like Google’s robots) while other robots cannot crawl (like OpenAI’s robots). 

In this example, from TechnicalSEO.com’s tool, the homepage is allowed to be crawled by Googlebot Smartphone. 

Crawling Tools

Crawling tools, like Screaming Frog, can also help you test these directives at scale. Use them to verify:

• Meta robots tags
• X-Robots-Tag headers
• Internal links to disallow or noindex pages
• Pages with noindex
• Pages with meta robots nofollow
• Rel qualifiers on outbound links

Log Files

For AI crawlers, there is not one standard version of Google Search Console. The better answer is log file analysis. 

After you disallow an AI crawler in robots.txt, check your logs to see whether that crawler is still requesting those URLs. If you add server-level blocking, check whether the bot is actually being blocked.

Recap: When to Use Noindex, Disallow and Nofollow

The biggest thing to remember is that these tools solve different problems.

• Disallow controls crawling.
• Noindex controls indexing.
• Rel=”nofollow”, rel=”sponsored”, and rel=”ugc” explain the nature of individual links.
• Meta robots nofollow controls whether bots follow links on a page, though this is rarely needed.

When in doubt, ask what you are trying to accomplish first. Then choose the directive that matches that goal. Common scenarios:

• If the page should not appear in search results, use noindex. You can use a meta robots noindex tag for HTML pages and the X-Robots-Tag noindex for non-HTML files. Do not block it in robots.txt if you need Google to see the noindex.
• If bots should not crawl the URL, use robots.txt disallow or server-level blocking. Remember that the URL may still be discovered, ending up in the index and ranking without content.
• If bots or users should never access the content, use authentication, IP allowlisting, or server rules. Do not rely on SEO directives for private content, including staging areas.
• If the page is duplicate or near-duplicate, find other solutions instead of noindex or disallow. A canonical tag might be the better fit or it might make sense to update the content to remove the duplication.
• If the page is low quality or outdated, do not start with robots.txt or noindex. Update it, consolidate it, redirect it, remove it, or noindex it only if the page still has a valid user purpose.
• If the link is paid, user-generated, or not fully editorial, use the right rel qualifier rel=”sponsored” (paid relationship), rel=”ugc” (user-generated content), or rel=”nofollow” (other, non-editorial link).

Finally, remember that you do not need to handle every bot or every page on your website the same way. You can disallow some pages or some bots, while allowing others.

Need Help?

If you have questions about any of these robot controls, or would like help working with these controls on your website, please contact me. For more information on noindex, nofollow, disallow, and other technical SEO subjects, please check out my book, Tech SEO Guide.

Resources

• Robotstxt.org – Guide for robots.txt
• Robotstxt.org – Meta Robots
• Google Webmaster Guidelines – link qualifiers
• Google Webmaster Guidelines – robots.txt
• Google Webmaster Guidelines – meta tag and X-Robot
• Google Webmaster Guidelines – control what you share
• Robots Meta Tags Supported by Bing
• Disallowing ChatGPT / OpenAI