Skip to content

Crawling

SEO begins with a robot crawling a website. During crawling, search engine robots open every page they have found a link to and fetch all the content from that page. You need to make sure robots are able to find all the pages and files on the website. As robots do find the pages and files, you also need to make sure robots can see the content on those pages and files. The resources below will help you find and fix crawling-related issues that are holding back your SEO performance.

Understanding TLDs

Understanding TLDs

How does top level domain (TLD) affect your website’s performance? Should you worry about what TLD your website uses?

SEO Discoverability

SEO Discoverability

Before you can do anything else with SEO, you need Google to find your pages. This process of finding pages is called “discoverability”. Learn about discoverability and how to make your pages easier for Google to find.

Robot Crawlability

Robot Crawlability

The most critical technical SEO task is making your website accessible to search engine robots that need to crawl through the various pages, images, and other content on your website.

XML Sitemap

XML Sitemap

An XML Sitemap lets you share data about the pages on your website with search engines.

Noindex vs. Nofollow vs. Disallow

Noindex vs. Nofollow vs. Disallow

There’s a common theme of confusion regarding the difference between noindex, nofollow and disallow commands. All three are powerful tools to use to improve a website’s organic search performance, but each has unique situations where they are appropriate to apply.

Selecting Canonical & Preferred Domains

Selecting Canonical & Preferred Domains

Should you use WWW or not in your URL? What about using an SSL certificate? In this blog post, learn what a canonical, or preferred, domain is and how this can affect your SEO and user experience.

Sub-Directory or Sub-Domain?

Sub-Directory or Sub-Domain?

Should you use a sub-directory (sub-folder) or sub-domain for SEO and for the best user experience?