Measuring AI Search: Options, Challenges, and What’s Next

As more people use AI search, companies are faced with a tricky question: how do we measure AI search? 

The biggest problem is that there is a lot of data we don’t have about AI search performance. As just a few examples:

• Google isn’t disclosing how many clicks come from AI Overviews or AI Mode. 
• OpenAI tags clicks so we can track traffic from ChatGPT, but they don’t share data about how often our companies are mentioned or cited in prompts. 
• AI agents, like Perplexity Comet, can visit websites without attribution. 

Without clear data about what prompts people enter, how often our sites are included in responses, or the traffic we get from AI responses, it is difficult to know how to invest in and optimize for AI search.

Let me be clear: AI search undeniably drives value. There is value in having AI discuss and recommend your products and services. There is value in AI agents using your website on behalf of a visitor to complete a task. In other cases, customers are using AI later in the sales process to summarize reviews and learn about a company as final decisions are made. This is a different type of value than what we’ve seen with traditional search, but there is still value from AI search.

The question, then, isn’t if AI drives value but how we effectively measure that value. The current options are often incomplete and imperfect. Let’s review the current measurement options by breaking this into two segments: what we can measure and what we can approximate. Along the way, I’ll also mention what we are unable to measure.

Quick Recap: Measuring AI Search

What We Can Measure

• Bot Crawl Tracking. Use log files or tools like Known Agents to see AI bots (e.g., ChatGPT-User) visiting your site, what pages they crawl, and how often.
• Traffic Attribution in GA4. For AI systems that pass referrer data (e.g., ChatGPT, Perplexity), filter GA4 traffic sources to measure visits, engagement, and conversions from AI-driven clicks.
• Bing Webmaster Tools AI Performance. Bing is the first company to release a tool sharing information about citations in AI responses, including how many pages were cited and how often.

What We Can Approximate

• Mentions in Prompt Responses. Use mention tracking tools to see if and how your brand appears in AI answers. Remember, results are based on example or synthetic prompts, not real prompts people are entering into Gemini, ChatGPT, Perplexity, and other AI platforms.
• Long Queries in Google. In Google Search Console, we can find longer queries people entered into Google search that may be more likely to generate an AI Overview.
• Indirect Traffic from Brand Mentions. Estimate AI-driven visits to our website within Direct traffic. For example, people may come to our website directly after seeing our company mentioned in an AI response instead of clicking a link.

What We Can’t Measure

• How often is your site included in actual AI answers
• How many people click on Google’s AI Overviews or AI Mode results
• The exact prompts users enter into AI systems and the exact response provided
• Interactions from AI agents (like ChatGPT Agent or Perplexity Comet).

What We Can Measure: Current Options for AI Search Tracking

Bot & Agent Tracking

AI platforms send out bots to crawl our websites. There are different types of bots and agents. Each uses our website in distinct ways. The primary types to consider when measuring performance:

• Training: Training bots are used to build AI system’s underlying models. These bots are crawling for deeper patterns about language usage and relationships.
• Search: Search bots are the most similar to Googlebot or Bingbot. These bots are fetching pages that can be listed in search results or citations.
• User interaction: These bots crawl a website whenever a user asks ChatGPT a specific question that requires live data from that website.

Each of these bots contributes a different type of value to your business. For example, training bot crawls indicate your website might be included in building the underlying model, giving your content a chance to influence how the AI system responds to prompts. Search bot crawls indicate your website may be listed in AI search results. 

The clearest value, though, comes from user interaction bots. These bots, like ChatGPT-User, indicate your website is being used to respond to a prompt in real-time. It is the closest data we have to how often your website is being discussed in AI responses. Two things to keep in mind: First, user interaction bots are not sent out for every request about your company. Second, just because it is sent out to your website does not guarantee inclusion in the response.

Agentic AI Interactions & AI Browsers

AI agents interact with our website on behalf of or alongside a visitor. We also have visitors using AI tools within their browser to help them visit our websites.

This data is not as easy to track. Many AI agents and AI-based browsers do not clearly identify themselves. For example, Perplexity’s new Comet browser doesn’t pass a unique user agent string. Without a unique user agent string, we are unable to tell if people are using Perplexity Comet to do something on our website. In log file data, when Comet uses our website on behalf of a person, it will look the same as a person coming directly to our website.

ChatGPT Agent from AI does identify itself using HTTP Message Signatures. Your web server can be configured to include these headers in access logs. You can also view HTTP Message Signatures to identify ChatGPT Agent in tools like Cloudflare, Akamai or Known Agents.

How to Measure AI Bots & Agents: Log Files & Known Agents (Dark Visitors)

Bot crawls and AI agent interactions can be measured through your website’s log files. Log files show specific bot hits, how often each bot visits your website, and what pages are crawled. For example, you can tell not only that OpenAI is using your website to train its models, but you can also see which specific pages are being included in that training data. With that data, you could update the pages OpenAI uses to train its models to make sure they accurately represent your company.

It is also important to assess what is not included in the log file. Which pages are not being crawled? Or, at least, not being crawled as much? These are pages that may represent bigger opportunities. By optimizing these pages or linking to these pages more, you may be able to get bots crawling these pages, allowing those pages to be included in AI responses.

I have a separate article providing details about how to access and use your website’s log files. Unfortunately, some hosting companies make it difficult to access the website’s log files. Even if you access logs, you may need to use expensive software to open up those log files (especially if you have a larger website). While log files are valuable, it can be high effort.

As an easier alternative, there is a WordPress plugin called Known Agents (formerly Dark Visitors). Set up is relatively straightforward. With this plugin installed, you can access much of the same data you can get through log files to understand what bots are crawling and what pages they crawl. For example, this screenshot shows the most active bots crawling a website, including AI bots ChatGPT-User and Meta-ExternalAgent.

Traffic Attribution in GA4

We can track if people click on links included in AI responses. AI platforms allow this either by passing referrer data or by tagging the links presented in responses. This is only true for some AI platforms. Currently, ChatGPT does pass this data. However, Google passes referrer data that looks like any type of search, preventing us from knowing if clicks came from AI responses or traditional blue links.

For AI tools that pass this data, we can attribute some of our traffic back to AI platforms. For example, we can see in our analytics tool how many visitors came from ChatGPT or Perplexity. We can also track if those visitors converted or engaged with our website. Even though the data is incomplete, the data we do get provides a fairly comprehensive look at how much traffic our website receives from people clicking on links contained in AI responses.

How to Measure AI Traffic: GA4

GA4 does not provide a built-in report to identify AI traffic. However, you can filter for this data under the Traffic Sources report. 

1. In GA4, go to Reports and then click on Traffic Acquisition. Traffic Acquisition is typically found under “Life Cycle” -> “Acquisition”.
2. Change the Primary Dimension from “Session primary channel group” to “Session source/medium”.
3. Above the graph, click “Add Filter”. In the filter panel, select “Session source” as the dimension. For “Match Type”, select “matches regex”. For value, enter the following text.
   
   (?i).*(copilot|chatgpt|openai|perplexity|gemini|claude).*
   
   This will filter the traffic data for any sources matching one of these AI services. You can adjust to add in more AI services depending on your specific needs and the AI systems you are interested in targeting.
4. The report will be updated to show sessions and engagement data for each source. The following screenshot shows the filter and the resulting table.

Bing Webmaster Tools AI Performance

Bing’s AI Performance report is available in Bing Webmaster Tools. It provides the first look at actual data about citations in AI search responses. The data is from Microsoft Copilot, AI-generated summaries in Bing, and selected AI partners.

In the graph, we have two data points:

• Total Citations. This shows you how many times your website was referenced or shown as a source in an answer generated by Copilot. In other words, how often was your website included in a response?
• Cited Pages. This tells you how many different pages were cited, not just how often they were cited. For example, you may have 83 total citations on a given day and there were 10 different pages included in those citations. The overall number shows the average number of citations per day.

In the table below the graph, we can see two additional sets of data:

• Grounding Query. These are not the prompts people entered that resulted in the AI response. Instead, these are the aggregated queries Copilot used to conduct its own searches when generating the prompt response. It can give us an idea of what the prompt topics might be of interest to people as it relates to our website.
• Pages: This shows you the specific pages on your website that were included as a citation in the prompt response. The data only shows total citations and does not indicate the ranking of the citation or how visible that citation was. It also doesn’t show if anybody clicked on the citation.

What We Can Approximate or Indirectly Measure

Mentions in Prompt Responses

Mention tracking tools track whether your brand appears in AI-generated answers. There are several different mention tracking tools available. Along with tracking if your company is mentioned in a response, these tools will also track how your company is mentioned. This is especially helpful in comparison to competitors: are competitors mentioned more often or in a more positive way than you are?

Unfortunately, there are several drawbacks to mention tracking. These systems use synthetic or sample prompts that people may enter. Right now, we do not know the actual prompts people are entering into ChatGPT, Perplexity, Gemini, Claude, or other tools. AI companies have not shared this prompt-related data. Mention tracking, then, is based on how closely those synthetic prompts align with what your real customers may be entering into AI systems.

While the data about prompts is limited, from what we know, AI prompts are highly personalized and unique. With heavy personalization, not only is every prompt different, but every response will be different as well. Responses are also unique to each person because AI systems factor in your history from previous chats. That means your website may be mentioned inconsistently based on the individual entering the prompt and how their prompt is worded.

That said, there are ways to get our example prompts closer to reality to make mention tracking more useful. A study reviewing leaked prompts found a few interesting characteristics:

• Prompts average 42 words, suggesting prompts offer lots of details.
• 75% of the prompts were not questions, but commands or tasks.
• Prompts are written assuming AI will understand the user’s context, including location.

If we can write our example prompts to match more of these characteristics, we likely will get better results from the mention tracking reports. It can also be helpful to ask your customers directly and ask them what types of prompts they entered related to your business. We’ve had a few clients do this and get real prompts from their customers. Seeing real-world prompts is valuable and then we can formulate other example prompts using that pattern.

In short, tracking mentions to a fixed set of responses can’t realistically model what LLMs are saying about your company. However, there are ways to improve mention tracking and get at least an approximate view of how your company appears in AI responses. Right now, this is the best way to understand if we’re showing up at all in some type of response and if we are showing up more or less than competitors.

Long Queries in Google

In Google Search Console, you can find long search queries. These longer queries typically take the form of questions or prompts and are likely to have triggered AI Overviews. While not an exact measure of AI Overview activity, it gives us a rough approximation of what prompts people may be entering.

To access this data, go to Google Search Console’s Performance report. Click “Add filter” above the graph and select “Query”. On the Query filter pop-up, choose “Custom (regex)” and Matches regex. Then enter this as the regex:

^(?:\S+\s+){8,}\S+$

You can change the 8 to another number to see longer or shorter queries. Once entered, click Apply. Scroll to the Queries table to see the updated table with longer queries. Typically, there is not much click activity for these terms. That’s okay because the value here is understanding the longer questions people are asking Google that are resulting in your website surfacing in a traditional ranking or in an AI response.

Indirect Traffic from Brand Mentions

A user may conduct a prompt related to your business and see your business mentioned. Even if there’s a link to your website included in the prompt response, people may not click on that. Instead, they may remember or copy your website URL and go to your website directly sometime later.

When people do this, they have found your website through AI Search, but your analytics tool will report this as a Direct visitor. That means some percentage of your traffic coming into your website that is labeled as Direct traffic is really traffic from AI Search. With this lack of attribution, we’re getting an incomplete look at how much value AI Search is driving to your website. That lack of attribution makes it difficult to know how you need to invest in AI Search and what types of things you did that resulted in customers finding your company in AI responses. 

There are some ways to work around this. You can review your website’s Direct traffic reports and try to attribute as much of the direct traffic as you can. For example, you may have sent out a newsletter on a particular day and you see a surge in direct traffic that day or the day after. In that case, the Direct traffic is likely from the email and less from other sources, like AI Search. 

Another way to evaluate traffic is to check how much of the direct traffic is coming into your website’s homepage. Direct traffic that comes into other pages of the website is likely misattributed traffic from other types of marketing. For example, you might have ads that weren’t tagged properly for some reason, and those will end up as direct traffic. Because you know that the direct traffic is going to an ad landing page, you can be reasonably certain that the traffic is actually from the ad. 

In the case of AI Search, when people come back to the website indirectly from a brand mention or a website mention, they will come back to the homepage. So in particular, you want to evaluate how many people came to the homepage from Direct traffic and if you are seeing any increases you cannot otherwise explain there.

If you cannot attribute the direct traffic increase to some other source, then at least some of that direct traffic may be coming from an AI Search result.

Operating with Imperfect Data

AI Search is here, but attribution is not. After years of data-driven marketing, this situation is tough. We need data to justify budgets and resources. If the data is unavailable or incomplete, how do we proceed?

Here are my recommendations:

1. Embrace Imperfection. The answer right now is to do as much as you can with the imperfect and incomplete data. Companies that succeed in AI search will be those that operate despite these limitations, instead of waiting for perfect data.
2. Educate Stakeholders. It’s crucial to help clients and stakeholders understand how to operate and invest without clear-cut data.
3. Prioritize Available Methods. We do have some data available to help.
   • Log files: Even if incomplete, this is the best data source we have available to get actual AI activity. With this, we can see what bots are doing on our sites and get a clearer indication of how to optimize our websites.
   • Mention tracking: Use it as the best available way to understand if your business is showing up in AI responses at all. Make sure your example prompts reflect reality as much as possible.
   • Analytics tools (GA4): Measure whatever AI traffic you can in your analytics system. Also, monitor Direct traffic to determine much direct traffic might be misattributed AI search.
   • Customer interviews: Interview a sample of customers to ask how they use AI search and what prompts they enter. Even from a limited sample, you can extrapolate potential broader customer behavior.

Need Help?

I am currently helping clients measure AI search performance and work within the limitations to understand how to optimize for visibility in AI responses. If you need help optimizing for or measuring AI search, please contact me.