What bot does Google use to crawl your website? You can find out in Google Search Console. Here’s how:
In Google Search Console’s sidebar, click on Settings, then click “Open Report” next to Crawl Stats (about halfway down the page). On the Crawl Stats report, scroll down to the “By Googlebot type” table. You’ll see the percentage of crawls per bot.
A few important considerations:
1) Chances are, you will see Smartphone or Desktop as the bot that crawls your website the most. That tells you what Googlebot’s primary crawler is on any given website. In some cases, Googlebot is primarily crawling, rendering and evaluating the mobile website, in other cases, it is the desktop website. Knowing this can change your design and development priorities.
It should also change what device you use to test your website. If Google mostly crawls via their Smartphone bot, you need to run your crawls and conduct your tests from a mobile device as well. This can help you identify files that are blocked for one device but not another. For example, JS or CSS may not load correctly on desktop but load correctly for Google on mobile.
2) This table also shows crawls from other Googlebot types. One of the more important types is “Page resource load.” This represents a secondary crawl of the website’s resource files, including CSS, JS or font files. These are the files needed to fully render the page. After the first crawl of the webpage, with the Desktop or Smartphone bot, Googlebot will run another crawl to fetch each resource file. (Note: These crawls can happen in different sequences and are not always aligned with the original page’s crawls.)
3) You will also see Image and Video listed here. Googlebot-Image and Googlebot-Video are specialty crawlers from Google used to understand the actual content of the video or image. Keep in mind: images and videos can also be fetched as part of “Page resource load” crawls. However, resource crawls are not attempting to understand the content of an image or video but, instead, are attempting to understand enough about the image to render the page.
For each bot type, you can click on the bot type to get a sample of the pages crawled on the website by that bot type and the trend line for how that bot crawls the website.
Take a look at your website in Google Search Console. What bot is Google primarily using to crawl your website? Are the various bot types crawling the appropriate files on your website?