Did GPTBot suddenly stop crawling your website? If so, don’t panic! That appears to be the normal trend…
GPTBot (OpenAI’s training bot) doesn’t crawl your website in regular intervals like Googlebot does. Instead, across every site I’ve seen so far, GPTBot crawls sporadically. It crawls heavily on some days, then is pretty much gone for weeks. Or it crawls one section of a website heavily on some days, then shifts to crawling another section.
So, if you see GPTBot suddenly stop crawling your website, or stop crawling a section of your website, it probably isn’t a problem. I get why clients keep asking me about this because the pattern definitely looks concerning. If you see it drop, definitely check to make sure nothing changed that caused that. But, more than likely, that is just normal behavior. More than likely, GPTBot will be back at some point in the future.
The heatmap below plots crawl activity for GPTBot across several different websites. The lighter colors indicate less crawling. The darker colors indicate more crawling, with red indicating heavy crawling from GPTBot.
A few things that really jump out to me:
- Every website has a different pattern, with many websites having many days where GPTBot didn’t crawl at all (the white squares).
- Heavier GPTBot crawls tend to line up on the same day across unrelated websites. This suggests OpenAI sends out GPTBot in bursts.
- This heatmap is a mix of big and small websites, and they all basically show similar trends. Big or small, GPTBot is not crawling in a regular pattern.
