Which bots matter the most for SEO and AI search or GEO?
Google uses different user agents for different purposes. Their primary bots for organic search are divided by device type:
- Googlebot Smartphone: This is the Primary search crawler for mobile devices.
- Googlebot Desktop: This is the Primary search crawler for desktop devices.
Google also has bots for specific content types:
- Googlebot-Image is used to crawl Images
- Googlebot-Video is used to crawl Videos
Google also has the Google-InspectionTool. This the bot used for Google’s testing products, like Rich Results Test and Inspect URL in Google Search Console.
There is also Google-Extended. This isn’t a real bot. Google-Extended is a name we can use on our robots.txt to control whether Google can include our website’s content in AI training. However, Google-Extended crawls use the regular Googlebot user agent.
Bing
Bing crawls the web similarly to Google, aiming to find content to rank in search results.
Bing crawls with the user agent Bingbot for organic search results. Unlike Google, Bing does not differentiate the user agent based on the type of content being crawled or by device.
OpenAI
OpenAI has multiple bots, each used for a different purpose.
- OAI-SearchBot is used to find pages to include in OpenAI’s search results. This is similar to Googlebot or Bingbot.
- GPTBot is used for training the model. If you don’t want your content included in training data, this the bot to block.
- ChatGPT-User is sent out in response to some prompts to fetch content from a website to possibly include in the generated answer.
Other AI
There are several other AI bots as well. A few notable ones to mention:
- Meta-ExternalAgent is Meta’s bot used for training for AI models.
- PerplexityBot is Perplexity’s main crawler for search results and training data. Perplexity also has Perplexity‑User to handle interactions on the site.
- ClaudeBot is Anthropic’s primary bot to collect training data. Anthropic also has Claude-User for site interactions and Claude-SearchBot to find pages to include in search results.
Recap
| User Agent | Company | Description |
|---|---|---|
| Googlebot Smartphone | Search crawler for mobile | |
| Googlebot Desktop | Search crawler for desktop | |
| Googlebot-Image | Image search crawler | |
| Googlebot-Video | Video search crawler | |
| Google-InspectionTool | Crawler for Google’s testing products (i.e. Inspect URL in Google Search Console) | |
| Google-Extended | Not a true bot, only use in the robots.txt file to keep content out of AI training data | |
| Bing | Bingbot | Search crawler |
| OAI-SearchBot | OpenAI | Crawler for search results |
| ChatGPT-User | OpenAI | User interactions |
| GPTBot | OpenAI | Collects training data for AI models |
| Meta-ExternalAgent | Meta | Training for AI models |
| PerplexityBot | Perplexity | Collects training data for AI models |
| Perplexity‑User | Perplexity | User interactions |
| ClaudeBot | Anthropic | Collects training data for AI models |
| Claude-User | Anthropic | User interactions |
| Claude-SearchBot | Anthropic | Crawler for search results |