Skip to content

What Are HTTP Headers

By Matthew Edgar · Last Updated: December 16, 2022

When a URL is typed into a browser (or when a search engine robot crawls a URL), a request is made to a website server. As part of this request, the browser or robot sends additional information to the server about the request in a text field. The server then sends back a file, such as a web page or image, in response to that request. Along with sending the contents file back, the server also sends additional information about the file requested in a text field. The text fields sent and received by the browser, robot, and server are referred to as HTTP Headers.

There are two main types of HTTP Headers: request and response.

  • Request headers are sent by the browser or robot to the server (as part of the request for a file). This includes sending information about the user agent (in the User-Agent header) or sending authentication information (in the Authentication header).
  • Response headers are sent by the server to the browser or robot (as part of the response to the request). Response headers include more information about the file returned, including information about the file’s status code or the file’s last modified date (in the Last-Modified) header. Some response headers can be classified as representation headers if the purpose of the header is to provide metadata about the file, like the file’s encoding.

Jump Ahead

How to View HTTP Headers

Viewing HTTP Headers In Chrome

View HTTP Headers in Chrome Developer Tools
View HTTP Headers in Chrome Developer Tools

HTTP Headers can be viewed in Chrome’s Developer Tools. After loading a page in Chrome, right click and select “Inspect”. On the Developer Tools panel, go to the Network tab and make sure it says “Recording network activity”. With the Network tab open, refresh the page in Chrome. The Network tab will now populate with all the resources required to load this page.

Click on the first item listed, which will be the URL of the page you are viewing. This will load a new panel with information about the page, including a tab in that panel containing the page’s HTTP Headers. Chrome Developer Tools will show both the Request and Response headers, which can be helpful for debugging any issues that might exist within the page.

Along with viewing the HTTP Headers for the main page, you can use Chrome Developer Tools to view the HTTP Headers for the other resources required to load this page. For example, it can be useful to view HTTP Headers for images, JavaScript, or CSS files to better understand how those files are loaded.

Viewing HTTP Headers in Google Search Console

View HTTP Headers in Google Search Console
View HTTP Headers in Google Search Console

You can view HTTP Headers in Google Search Console using the URL Inspect feature for any page with a status 200 (which is any normal page that does not redirect or return an error). In your Google Search Console account, enter any URL on your website into the search field at the top of the page. This will load the data Google has about that URL if the URL has already been crawled by Google. If no information is found for this URL, you can click Test Live URL to have Google fetch the page.

Once the URL Inspect page loads, clicks “Viewed Crawled Page” (or if you have run a live test, click “View Tested Page”). On the panel that expands, click the “More Info” tab, then click “HTTP Response”. This will show the HTTP Headers Google received on the latest crawl of the page, which can be interesting to check against the live HTTP Headers seen on the site.

Along with showing the HTTP Headers found on the latest crawl, Google will also add additional information to the headers providing a bit more context about how Google has crawled the page. This includes X-Google-Crawl-Date, which shows the last date Google crawled the page.

SEO Specific HTTP Headers

There are also SEO-specific HTTP Headers that can be used to define robot directives and to define the page’s canonical tag. These headers can be used in place of the HTML tags and are often used on non-HTML content, such as PDF files. Note that these headers are only used by Googlebot and Bingbot but will not be used by any other robots or browsers.

X-Robots-Tag

The X-Robots-Tag header operates like the meta robots tag, defining how links contained within the file should be crawled (follow or nofollow) and if the file should be indexed (noindex or index). Other directives can also be specified in the X-Robots tag, including whether Google is permitted to create a snippet for the file (via the nosnippet or max-snippet directives) and whether images contained in this file can be indexed (via the noimageindex directive).

The X-Robots-Tag is typically defined in the .htaccess file on Apache servers. For example, this code would apply a noindex via the X-Robots-Tag header to every PDF file on the website:

<files *.pdf>
Header set X-Robots-Tag "noindex"
</files>

As another example, this code in the .htaccess file would set a noindex for the file example.pdf contained in the /uploads/ directory.

<If "%{REQUEST_URI} =~ m#^/uploads/a-specific-file.pdf#">
Header set X-Robots-Tag "noindex"
</If>

Link rel = “canonical”

The rel = “canonical” HTTP Header works exactly like the canonical specified in the element for Googlebot. It is not clear if Bingbot supports this header. Whether specified in the HTML head or in the HTTP Header, the canonical tag helps to resolve duplicate content by defining a canonical or preferred URL.

The rel = “canonical” HTTP Header is defined with the Link HTTP Header, which requires a URL and additional parameters (in this case, the parameter will note that this link contains the canonical URL). This is typically defined in the .htaccess file on Apache servers. For example, this code in the .htaccess file defines Elementive’s about page as the canonical URL of this document.

<If "%{THE_REQUEST} =~ m#about-elementive.docx#">
Header set Link '<https://www.elementive.com/about-elementive/>; rel="canonical"'
</If>

Here is an example of the headers for this doc file in Google Search Console, including the Link containing the canonical in the HTTP Headers.

HTTP Headers in Google Search Console, showing the Link rel="canonical"
HTTP Headers in Google Search Console, showing the Link rel=”canonical”

Common HTTP Headers

There are many HTTP Headers (view the full list at MDN), but there are a few that are of particular importance for SEO.

Location

The Location HTTP Header is required for any redirect. This is where the server indicates where to redirect the visitor. For example, in the screenshot below, the Location HTTP Header tells robots and browsers that the requested URL (https://www.matthewedgar.net/technical-seo/) has been moved to a new URL (https://www.matthewedgar.net/tech-seo/).

HTTP Headers a redirect indicating the Location
HTTP Headers a redirect indicating the Location

Retry-After

When a temporary server error occurs, the server should respond with a 503 response code. Because the 503 response is a temporary error, the server should communicate when a browser or robot should retry accessing the page. This can be done in the Retry-After response header. The same is true for a 429 response code, indicating too many requests have been made and the browser or robot making those requests must wait some interval of time before requesting again. The Retry-After should be set to a date or a number of seconds to wait.

Content-Encoding

One of the best ways to improve a website’s speed is to compress the website’s files. However, a browser or robot will need to decompress (or decode) those compressed files in order to appropriately read the file. As a result, the server needs to tell browsers or robots requesting files how those files have been compressed (or encoded). This is done with the Content-Encoding HTTP Header. Two of the more common encoding methods are Gzip (represented as gzip or x-gzip) and Brotli (represented as br).

Related to Content-Encoding is the Transfer-Encoding header. You’ll sometimes see the Transfer-Encoding header in Google Search Console. The Transfer-Encoding header also indicates what encoding the server has used but the Transfer-Encoding is hop-by-hop, meaning it indicates encoding used through each step (or hop) of the connection. Content-Encoding is end-to-end, meaning it indicates encoding used across the entire connection.

Cache-Control (and Expires)

The Cache-Control header defines how, or if, the resource should be cached. Caching a file helps decrease the website’s load time, helping the website load faster.

There are multiple ways this header can be set. If a file should not be cached at all, the Cache-Control header can be set to no-store. The Cache-Control header can also be set to no-cache, which does allow caching but only if the browser checks for updates first before using the cached file.

If the file is cached, it is important to tell the browser how long the resource should be cached. This is done by defining max-age in the Cache-Control header. For example, the font file shown in the image below is cached with a max-age of 31,536,000 seconds (one year). This tells browsers that the font file can be kept in the cache, and not re-requested from the server, for up to one year. Importantly, Google has indicated that Googlebot does not use this header.

Example of Cache-Control HTTP Header
Example of Cache-Control HTTP Header

The Cache-Control header was defined in HTTP/1.1. Previously, the cache timing was defined in the Expires header. The Expires header specified the date after which the browser should consider the file to have expired; this allowed browsers to cache the file up to that date. In the screenshot above, the Expires header is set to one year from the date the resource was first loaded (matching the max-age specification). If a Cache-Control header is set, then the Expires header will be ignored but it can be helpful to specify the Expires header since older browsers may not fully support the Cache-Control header.

Content-Type

The Content-Type HTTP Header tells browsers and robots what type of file is being delivered. It is from this information that robots and browsers know if a particular file is an image, an HTML document, a JavaScript file, a CSS file, or a video. The screenshot below shows the HTTP Headers for the logo images on MatthewEdgar.net, which uses a WebP image file type, and Elementive.com, which uses a PNG image file type. CSS files will show “text/css” and JavaScript will show “application/javascript”.

HTTP Headers for images showing different content types
HTTP Headers for two images showing different content types (WebP and PNG)

Content-Language

When requesting a page, browsers or robots can optionally send an Accept-Language request header. Severs can then respond with the Content-Language header that indicates the selected language. While most browsers support this header, Googlebot does not crawl with accept-language headers. For SEO purposes, different page languages should be specified with hreflang tags. However, it is best practice to have the Content-Language header, if used, set to the appropriate language for each page to avoid any confusion about what content is contained on the page.

Final Thoughts

It is important to review what HTTP Headers are returned on your website and to understand what those HTTP Headers are communicating about your website’s pages to browsers and bots. If you need help reviewing the HTTP Headers on your website or need help with other technical SEO issues, please contact me.

You may also like

How to Use a Headless Browser

Learn what a headless browser is, why you should use a headless browser and how to launch a headless browser on your own computer.

SEO Friendly Infinite Scroll: How Do You Make Infinite Scroll Work For Organic Search?

How do you make infinite scroll and related techniques SEO friendly? See the steps you need to take to fix infinite scroll for SEO.

How to Check HTTP Response Status Codes

Every page on every website returns an HTTP response status code. How do you check the status code for your website’s pages? What do the status codes mean?