AI search creates a measurement problem: AI search tools may crawl your website far more often than they send visitors back to it. How does this change measuring SEO performance?
Cloudflare introduced a metric to understand this measurement problem called the crawl-to-refer ratio to compare how often a bot from a given platform crawls a website against how much referral traffic that same platform sends back.
Per Cloudflare’s data, Google crawls about 5 pages for every 1 referral visit it sends. Meanwhile, OpenAI crawls 1,100 pages for every 1 referral visit.
That is a big difference. At first glance, it appears that OpenAI simply isn’t driving any value. So, it is understandable why website owners would look at that and ask: “Are we getting enough value?”
What this really gets to is exactly how AI search is different from traditional search. With Google, traffic was pretty much always the main way we got value. Google crawled our site, ranked our pages, and sent visitors back.
With AI search, referral traffic is only one possible way we can get value. Value might also show up as:
- Brand mentions
- Citations in AI answers
- Direct calls or emails
- Branded searches later
- Social follows
- YouTube views
- Offline conversations or conversions
How valuable are any of those things relative to crawls? That is the question every business has to figure out right now. The answer will be different for every business.
The crawl-to-refer ratio is useful because it helps you understand the relationship between crawl volume and referral traffic. I don’t think it is bad, but it isn’t enough by itself.
We also need to consider the ratio between crawls and the other ways AI tools drive value. For example, what is the ratio between crawls and citations or the ratio between crawls and unattributed conversions? There aren’t easy ways to measure those numbers right now and these are harder to quantify.
In short, I wouldn’t use this metric and the lack of traffic alone to decide whether a bot should be blocked or allowed. There is a lot more to consider about what value AI platforms create for the business and how many crawls that is worth.