Indexing is the process of storing crawled and rendered content in a structured way so it can be retrieved during search or response generation.

SEO is no longer just for Googlebot. This post is part of a series on adapting technical SEO for AI bots and new search behaviors. Explore the full series.

Why do some pages get crawled but not indexed?

Google’s index is used to populate search results. Because of that, Google only wants to index pages that are worth ranking. More precisely, Google wants to rank pages that appear to be relevant to potential search queries and people. 

Not all pages are worth ranking, at least according to the evaluation of Google’s bots. If the bots determine a page contains low-quality content, outdated content, or duplicate content, it will not be indexed. A large part of SEO is making sure that Google correctly evaluates the website’s content, so that the content is correctly indexed.

Unfortunately, Google’s bots do not always make an accurate evaluation of the page’s content quality and keep it out of the index as a result. In many cases, this incorrect evaluation is caused by a crawling or rendering error. If bots are unable to successfully crawl or render a page to access all of the page’s content, bots will make an indexing decision based on an incomplete understanding of the page.

During indexing, Google’s bots also detect errors, including not-found errors and soft 404s, and keep these pages from being indexed. So long as the errors are valid, then not indexing these pages is the correct choice. However, Google’s bots may mistake certain types of pages for errors. For example, a directory may have several entries with only limited content and Google’s bots might flag these as Soft 404s.

How does AI index content differently from search engines?

AI search engines do not use an index in the same way traditional search engines use an index. In traditional search, the search engine queries the index for items related to the query and then outputs an ordered list of those items. The index is central to the process. This is why so much of our SEO practices focus on getting pages indexed.

AI search engines (like Google’s AI Mode, Perplexity, and ChatGPT) store content in a knowledge base (Google’s search engine is likely the foundation of this knowledge base for Google’s results). When responding to a prompt, the AI system will query for items similar to the prompt. Note the word similar. When retrieving items from the knowledge base, the focus is on finding items with similar meaning and intent, not just finding items with the same words. This is a large part of why keyword optimization is no longer effective. Content now needs to consider topics and intent, rather than just keywords.

Once retrieved from the knowledge base, the system will pass those similar items along to the Large Language Model (LLM). The LLM may use these items to generate a response and may include some of the returned items in citations. That is, listing items from the knowledge base in a response is optional, and the LLM will make decisions about what to include in real-time when generating a response.

In short, with traditional search, the index was central but in AI search, the LLM is central. It is important to remember that an LLM is not an index. An LLM is not a structured representation of information. Instead, an LLM uses training materials to learn language patterns and relationships between words. As a result, how we optimize for inclusion in an LLM is different from how we optimize for inclusion in the AI system’s knowledge base.

Learn more about the differences between traditional search and generative AI search.

What is passage-level indexing?

Traditionally, search engines indexed the whole page. The index was organized around pages and included additional information that bots knew about each page. If the entire page was not relevant to a search query, then the page would not rank or at least not rank well, even if a part of the page was relevant to the query.

Starting in 2020, Google began shifting towards passage-level indexing. Instead of indexing whole pages, bots now index chunks of content from a page. This means that parts of pages, like individual sentences or paragraphs, may rank for a search query even if that part is a smaller portion of the page. For example, a poorly structured FAQ page covering a wide range of topics might not rank in traditional search, but could still be cited in an AI search response due to a semantically rich paragraph.

AI engines also chunk pages into passages. When an AI system is querying the knowledge base for similar items, it is not looking for similar webpages. Instead, it is looking for similar sentences or paragraphs. This is why longer pages about a wide array of topics can still be listed as a citation even if most of the page is not highly relevant.

Technically, passage indexing makes using proper HTML structure even more important. Using headers (h1, h2, h3, etc.) appropriately can make it easier for bots to identify and extract sections of a page. Semantic HTML (<article> vs. <div> tags) may also be useful to assist bots in chunking down page content. Adding jump links within longer pages also allows bots to link directly to specific sections of a page.

Schema markup, or structured data, may help with passage indexing. With schema, you can clarify and explain content more precisely than possible with basic HTML. For example, I can explain that a list of items on a page is a list of local businesses or a list of products. Many SEOs are actively experimenting with this, including me. Early signs are that schema may help improve passage (and page) relevance.

How do you monitor indexing?

For Google search, the only way to monitor indexing is Google Search Console’s Page Indexing report. This report shows details about which pages are indexed and, more importantly, which pages are not in the index. It will also provide details about why some pages are not indexed. I have a separate article digging into Google’s Page Indexing report.

Bing provides a similar report in Bing Webmaster Tools. Bing’s Site Explorer provides a hierarchical view of the website. For each page and section, Bing will show information about indexed pages. Using the “Filter by” dropdown, you can show pages with various types of problems.

Bing Webmaster Tools Site Explorer

So far, OpenAI, Perplexity Meta, and other AI tools do not provide any type of reporting about what pages are included in the knowledge base or what pages are used for training. The only way to determine what pages these systems know about is to ask the engines directly with carefully constructed prompts. For example, asking the engine what it knows about a specific subject covered by your company on your website could potentially lead to your website being included in citations. This is an imperfect way to assess inclusion in the index, training data, or the knowledge base.

Key Takeaways: Indexing

  • Google only indexes pages it deems high quality and relevant to search queries. Pages with low-quality, duplicate, outdated, or inaccessible content often will not be indexed. 
  • Errors in crawling or rendering can also prevent good pages from being indexed. An indexing problem may require investigating the page for technical problems, not just content problems.
  • Modern SEO and AI systems store and retrieve information based on semantic similarity, not exact keywords. Optimization for current search and AI systems involves focusing on intent and meaning, rather than just keyword usage.
  • Both traditional and AI search systems now focus on indexing passages (paragraphs or sentences) rather than indexing full pages. This means individual sections of a page can rank or be cited even if the rest of the page isn’t highly relevant. Using structured HTML and schema markup helps bots accurately identify and extract these relevant parts.

More on Modern Bot Behavior

This article is part of my series on how AI search is changing technical SEO and breaks down how we optimize for the six modern bot behaviors.

Read the next part: Training: How your site content feeds large language models

Need help understanding how AI bots are using your website or want to optimize for AI search? Contact me and let’s talk.