How to Use Wildcards and $ in Robots.txt

Matthew Edgar | August 31, 2026

The robots.txt file supports the use of special matching characters. These can cause some problems if they are not used correctly…

The robots.txt file supports two types of matching characters:

* - matches any character
$ - states the end of the URL

For example:

User-agent: *
Disallow: /specials*

Will disallow any URL that starts with specials. That could match:
/specials
/specials/
/specials-2
/specials/page.html

If your intention is to block that many URLs, the rule is correct. However, it is more likely you wanted to be more selective in what gets blocked. I’ve seen a lot of sites end up blocking pages they didn’t mean to because of the * being used incorrectly.

That said, the wildcard match can come in handy if you want to block a whole type of file. For example, if you want to block all PDFs from being crawled:

User-agent: *
Disallow: /*.pdf

The $ matches the end of the string. It isn’t as commonly used. Where I’ve typically seen it is when you want to allow crawls of the homepage but disallow other pages to get crawled:

User-agent: *
Allow: /$
Disallow: /

There can be some problems though with this character. I’ve seen people try to block PDFs with:

Disallow /*.pdf$

That disallow would mean that only URLs to a PDF that end in .pdf would be blocked from crawling. If you pass a parameter on the URL (such as tracking parameters), then crawling would be allowed. With that Disallow rule, the URL /some-file.pdf?utm_source=ads would still be crawled and only /some-file.pdf would be blocked.