URL parameters, also known as query strings, are part of a webpage’s URL. Parameters can provide additional information about the specific content being requested. For example, a parameter may be added to a URL to indicate the page’s content is sorted or filtered.

Parameters follow a question mark (?) in the URL and are composed of a key-value pair separated by an equal sign (=). Multiple parameters can be added by separating them with an ampersand (&). For example, in the URL https://example.com/page?param1=value1&param2=value2, param1 and param2 are URL parameters with values value1 and value2, respectively. The entire text after the question mark (param1=value1&param2=value2) is the query string.

SEO Challenges Presented by URL Parameters

URL parameters can present challenges for AI and search bots when crawling a website, including:

Duplicate content: URL parameters can create multiple URLs that display the same or very similar content, which results in these pages being considered a form of duplicate content. For example, mysite.com/product-list and mysite.com/product-list?sort=price would contain the same content, just sorted in a slightly different order. This type of duplication can be a form of low-quality content, which will not perform well in rankings or citations.

Crawl budget: Bots can only crawl so many pages on a website at any given time. When there are multiple URLs with similar content due to URL parameters, bots may waste time crawling irrelevant pages instead of discovering new and more valuable content.

Internal link signals: If bots see multiple versions of the same content with different URL parameters within internal links, it can lead to confusion about which URL should be considered the canonical URL that should rank in search results. For example, the canonical URL might be declared as mysite.com/product-list, but bots might ignore that canonical URL if all internal links point to mysite.com/product-list?sort=price.

Manage Parameters Effectively to Avoid SEO Problems

In this article, let’s review the best ways of managing parameters on a website for the best performance in traditional search rankings and citations in AI responses. This breaks down into the following steps:

Step #1: Find Every Parameter on Your Website

Before you know how to manage parameters on your website, you’ll need to first find every parameter on your website. Realistically, you won’t find every parameter, but you can find the most common and most problematic. This includes parameters generated by your content management system, parameters generated by third-party tools used on your website, and tracking parameters used within advertisements and promotions.

Ideally, you could ask the marketers, developers, and partners at third parties for a list of each parameter used. This isn’t a bad step and can provide a starting point. In reality, though, companies rarely keep a complete record of each parameter used and, in other cases, can’t keep a complete record since many parameters are dynamically generated.

There are many places you can look for parameters, but here are three of the places I’ve found to be the most helpful: log files, analytics, and site crawl tools.

Log Files: Find Parameters Google, Bing, OpenAI, or Other Bots Have Crawled

We can find URLs with parameters in log files that bots have already crawled. Log files track the files all visitors, including bots, requested on a website. This includes URLs bots may be crawling with parameters. Learn more about using log files for SEO analysis.

If you do not have access to your website’s log files, you can also find URLs that might be crawled in Google Search Console’s Crawl Stats report. This only shows you the crawls by Google, and no other tool, but it is at least a means of seeing what URLs with parameters may exist on your website. Here is a video about the Crawl Stats report:

Web Analytics Tools: Finding Parameters in GA4

Web analytics tools will tell you which pages with parameters have been visited on the website. In GA4, this can be found in the built-in “Landing pages” report. On the “Reports” page, go to the “Life cycle” section, expand “Engagement”, and then click “Landing page”.

Once on the report, add a secondary dimension by clicking the blue + icon in the table header row. Select the secondary dimension of “Landing page + query string”. Then search for any URL containing a question mark. This will show the URL with parameters. In the upper right corner, click the share icon and then click “Download File”. This file can then be downloaded to Excel or Google Sheets. Once downloaded, you can extract the parameters from the URLs (more details on extraction in Step #2).

GA4 Landing page + query string

Finding URL Parameters in Crawl Tools

Finally, you can use a crawl tool to locate parameters that are included within internal links on the website. We can see an example of this in JetOctopus. After running a crawl, view all links found on the crawl. Then, filter for any links that have a link destination containing a question mark. To avoid finding external links that happen to contain parameters, you can also add a condition where link destinations must include your website’s address. Here again, this list can then be exported to Excel or Google Sheets where parameters can be extracted from the URL (more details on extraction in Step #2).

Finding URLs with query strings in Jet Octopus

Step #2: Extract Parameters from URLs

Once you have the exported lists, the next step is getting a complete list of parameters contained within those URLs. Begin by combining all of the URLs listed in the exported files together. That way, you have a complete list of all URLs with parameters that are used on your website.

To help you extract parameters, I have a free bulk parameter extraction tool you can use. Using this tool, simply paste in a list of URLs with parameters and then click the “Get Bulk Parameters” button. The page will then show a list of all the parameters used along with their respective values. Using this, you can start to review what parameters are and aren’t needed on the site, which I’ll cover in the next step.

free bulk parameters tool

Step #3: Evaluate Parameters to Identify Problems

Once you have extracted the parameters from the URLs on your website, the next step is determining if those parameters are necessary, how to handle those parameters, and if any parameters are presenting problems. If those parameters do present problems, you want to figure out how to fix. There are a few key questions to address within your evaluation:

Does the parameter meaningfully change the page’s content?

Many parameters can change the page’s content, but the question is if that change is meaningful to visitors. The more meaningful the change to the content, the more that parameter is something automated visitors need to see on the website.

Take, for example, a parameter that sorts the page differently, such as ?sort=alpha or ?sort=low-price, which sorts a list of products alphabetically or by lowest to highest price. That sort of functionality is helpful for people who are already visiting the page. However, resorting the list doesn’t meaningfully alter the page because it is the same list of products. Somebody searching on Google may want to find the list of products but won’t be so interested in finding that list sorted one way or another. An AI response, though, may alter what URL is cited based on the sort.

As another example, there might also be a parameter that filters that list of products. By adding ?filter=blue to the URL, the list of products will only show blue products instead of showing all products. This is a more meaningful change to the page’s content, both for visitors who are on the page and for people seeking out this page in search results or AI conversations. People looking for blue products would likely be interested in seeing the blue products page in rankings or citations instead of the page that shows all products.

With other parameters, though, no changes are made to the website’s content at all. Tracking parameters, like UTM parameters, can be added to URLs that identify a particular traffic source and pass that information along to an analytics program. Because these parameters do not alter the website’s content, they are likely not something bots need to crawl.

Should URLs with this parameter be part of the search index or AI training?

There are some parameters that should absolutely not be included in a search index or be part of AI training material. Tracking parameters are a prime example. Let’s say a page containing a tracking parameter associated with an ad campaign is cited in an AI response. If somebody clicks on that citation, your analytics tool will incorrectly report the visit’s source as the ad campaign instead of as a visitor from the AI platform.

If the parameter alters the page’s content in a meaningful way, it is usually best to let bots crawl, index, and use that page for training data. Because the parameter changes the content in a meaningful way, the page offers something meaningful that can help bots understand your website. For parameters that don’t meaningfully change the content, those should often be blocked from crawling or indexing.

Figuring out what to allow may vary depending on the bot in question. Let’s take sorting parameters as an example, because context may change whether sort parameters are a meaningful change to the page’s contet:

  • When it comes to traditional search results, sorting may not be a meaningful change. Allowing Google to rank the page sorted in any direction (ranking the default sort, the alphabetical sort, and the price sort) could create duplication problems where these pages compete for the same search terms. In these cases, you need to control which parameter is ranked and block indexing of sort parameters for Google.
  • However, the sort parameter may be meaningful for an AI response. If somebody asks ChatGPT for the cheapest items on a retailer’s website, the AI would likely benefit from training against a page sorted from low price to high price. The page sorted by price may also be considered relevant enough by the LLM that it is mentioned in the response or mentioned in the citations. In that case, you may want to allow ChatGPT to crawl the page and use it for training.

If you aren’t sure, the default answer should be to let bots crawl pages and allow bots to index those pages.

Along with considering how meaningful the content is, remember that crawling so many URLs with parameters may overload the server. In those cases, it makes sense to block bots from crawling the page regardless of how meaningful the page might be for rankings or citations.

Step #4: Control and Manage Parameters

After evaluating your website’s parameters and considering each of the questions presented in Step #3, there are three different methods for controlling and managing parameters.

  • Allow Crawling, Indexing & Training. If the parameter meaningfully alters the content, bots should be allowed to crawl, index and use the pages for training.
  • Prevent Ranking & Citations. If the parameter does not alter the content, that page should not be indexed, crawled, or used within training data.
  • Block Crawling. If there is a problem with bots crawling too many URLs containing a parameter, the bots can be blocked from crawling any URLs containing that parameter. Decisions do not have to be universal and can be made for specific URLs and specific bots.

Allow Crawling, Indexing & Training

For pages that you do want bots to crawl, index, and use for training data, the page with the parameter should have a unique title, meta description, and H1 tag. As well, other text on the page should be at least somewhat unique to make that a distinct, meaningful page on the website. Finally, the page with the parameter should contain a self-referencing canonical.

As an example, consider an e-commerce website that has the main product list available at the URL mysite.com/product-list and a filtered product list available at a URL with parameters, mysite.com/product-list?size=large. Both pages could contain meaningful information and offer a different list of products. If the title, H1, and other text on the filtered product list page are all identical to the title, H1, and text on the main product list page, then these pages will be seen as duplicate content. That duplication may prevent the pages from being indexed or from being used in training data. It may also prevent one or both pages from ranking or appearing in citations.

It is also important to make sure internal links point to the page with the parameters included in the URL and, potentially, list the URL with those parameters in the XML sitemap. For similar reasons, you could also build external backlinks to the URL containing that parameter.

Prevent Ranking & Citations

If the parameter does not meaningfully change the page’s content, you want to keep this page from ranking in search results or being used within AI citations. These are controlled in different ways.

To prevent indexing, there are two primary methods available: canonical or noindex.

  • On a page with a parameter, you can define a canonical tag and have the canonical point to a version of the page without that parameter. For example, https://www.site.com/test?sort=alpha might canonical to https://www.site.com/test instead. This would indicate to Google and Bing that you consider the page with the parameter to be a duplicate, or a near duplicate, of the page without the parameter.

    In most cases, Google will index the canonical version of the URL that does not include the parameter. However, it is important that internal links support the canonical choices as much as possible to reinforce what the canonical tag is communicating to Google. In the previous example, if /test were selected as the canonical, internal links to /test?sort=alpha should be minimized.
  • As an alternative, a meta robots noindex tag can be used on any page containing a specific parameter. This is a stronger signal than a canonical and instructs Google and Bing to keep the page with that parameter out of the index and out of rankings. Typically, this method should only be considered if Google is not respecting the canonical tags.

To prevent training, the best answer is to block crawling with disallow statements referring to specific AI training bots on the robots.txt file. Details about blocking crawling are discussed in the next section.

Block Crawling

The easiest way to prevent bots from crawling a page containing a specific page is to use the disallow command on the robots.txt file. This instructs bots to ignore any disallowed page when crawling the website. Internal links should only minimally use the disallowed parameter paths to further discourage robots from finding and crawling the pages. Here is an example of a disallow statement blocking crawls of the “test” parameter:

Disallow: *test=*

The disallow is optional for robots to follow, though most major AI and search bots will respect the robots.txt file. If the parameter appended should never be crawled by any robots, stronger measures need to be taken. For example, consider a parameter that exposes a development environment. In these cases, you want to carefully control who sees the development environment and who doesn’t. A disallow wouldn’t be a strong enough signal to ensure robots don’t crawl the development environment pages. A better option is to restrict access to only certain IP addresses or require people to log in before accessing the page with that parameter. Learn more about handling development environments for SEO.

This works well to prevent AI bots from training on the disallowed pages. However, there is a problem when it comes to indexing.

For search engine bots like Googlebot and Bingbot, blocking crawling (regardless of the method used) only prevents the bot from crawling the page. It does not necessarily prevent the page from being indexed or appearing in search results. If bots find the URL with that parameter in some other way, the page may still end up in search results. This is because the noindex directive, which instructs Google and Bing to keep the page out of rankings, is included on the page itself. So, if Google or Bing can’t crawl the page, they can’t see the noindex. You can end up with pages that are indexed even though bots can’t crawl the page. Google will state that “No information is available for this page” in the ranking:

Disallowed page ranking in Google search results

This is part of why it is important to remove internal links to pages that should not be crawled. If internal links must be added, it is generally best to allow search bots to crawl pages with parameters so that you can control the page with a noindex tag.

If you must prevent crawling, the other way of keeping the pages out of rankings is with Google Search Console’s removal tool. You can state the parameter as part of the prefix. For example, removing URLs with the prefix https://www.test.com/some-url/?test= would fix the indexed example above. Here is a video discussing the removal request tool in more detail:

Final Thoughts

Parameters present a number of challenges for SEO and AI search. URLs with parameters can lead to duplication, which makes it harder for all pages on the website to rank or be included in citations. Parameters can also cause bots to crawl unnecessary pages and miss the important pages on the website.

It is important to review your website to see what parameters are used and what changes you need to make to handle parameters effectively. The process can be broken down into a few main steps: finding, extracting, evaluating, and controlling parameters. To find parameters, check your analytics, log files, and conduct a site crawl. When evaluating parameters, consider if they meaningfully change the page’s content, if they should rank in search results, and if Googlebot should crawl URLs with the parameter. Finally, you want to carefully determine which parameters bots should be allowed to crawl, index, and use within training data. By following these steps, you can effectively manage your website’s parameters to improve SEO and GEO performance.

If you have questions about managing parameters on your website, please contact me. For more help with technical SEO, you can also check out my book, Tech SEO Guide, which provides a reference for managing parameters and many other aspects of the technical side of SEO.