One key metric we have to understand AI search is bot crawl volumes. How often do ChatGPT, Perplexity and Google crawl our website? What pages are they seeing? What are they missing?
If AI bots are crawling your website, then that is the first step to being included in responses. So, tracking bot activity is a direct measure of successful AI optimization.
Tracking bots also helps you:
- Spot crawl issues
- Find error pages
- Evaluate redirects
- Prioritize pages for optimization
- Identify low-value content to remove or block
The best way to get this data is by reviewing your website’s log files. Accessing logs through your host isn’t always straightforward, and larger sites often need expensive tools to parse the data. It’s valuable, but not always easy. So, a lot of companies end up not using log file data.
If you’re running a WordPress site and want a simpler option, there is a free plugin called Known Agents (previously called Dark Visitors). It offers an easier way to monitor how search engines and AI bots crawl your site without needing to dig through server logs.
I recorded a video going over how Known Agents works in more detail. Note, the video was recorded when the tool was called Dark Visitors but the concepts remain the same!