Discovery is the process by which automated services (including search engine crawlers, AI bots and AI agents) identify new URLs or content for the first time. Before you can do anything else with SEO and AI Search, Google, Bing, OpenAI, Perplexity, Claude and others need to discover the pages on your website. If they do not find every page on a website, the site will struggle to perform in organic search.

SEO is no longer just for Googlebot. This post is part of a series on adapting technical SEO for AI bots and new search behaviors. Explore the full series.

What signals help bots discover new URLs?

The primary source of discovery is hyperlinks, both internal (links on your website) and external (links from other websites). When bots find a link within a web page’s HTML, they extract the URL contained in the standard HTML anchor tag (<a>) and its href attribute, adding that URL to their database and crawl queue.

However, not all links will encourage discovery equally:

  • Prominence and Quantity: Links on more prominent pages, such as your homepage, or from popular external sites are picked up more quickly. While bots can discover a page from a single link, having multiple internal links referencing a page greatly improves the chances of discovery.
  • Link Qualifiers: You can add rel attributes like rel=”sponsored” or rel=”ugc” to communicate the context of a link to Google’s bots, allowing them to handle the link appropriately after discovering it. Learn more about rel qualifiers.
  • Feeds: RSS feeds are regularly crawled by Google, Bing and some AI bots, sometimes more frequently than other pages, making them an excellent discovery point, especially for news sites. AI tools are also using product or forum feeds to gain direct access to content.
  • XML Sitemaps: An XML sitemap is a list of all pages you want Google to crawl. They are a particularly helpful solution for large websites or sites with technical constraints that make internal linking difficult. Just ensure your sitemap is accurate; do not list error pages, redirects, or noindexed URLs.
  • HTML Sitemaps: While bots crawl them unreliably, HTML sitemaps can still be an effective discovery tool for complex websites with dense hierarchies.
  • LLMS.txt: This is an emerging standard that is not widely supported. However, LLMS.txt files can provide a list of URLs on a website and may help AI bots discover URLs on a website.

What causes discovery failures?

Adding a link to a page isn’t enough; bots must be able to crawl the code and successfully extract the links. Discoverability fails for a few common reasons:

  • Client-Side JavaScript Links: Websites that rely heavily on JavaScript to construct pages or add links after a page loads (e.g., in response to user interactions) create massive problems for discovery. If bots cannot successfully execute the JavaScript, they won’t see the links. To maximize discoverability, place links directly into your HTML using server-side code. (Learn more about how JavaScript affects SEO and AI search.)
  • Improper Link Tags: Broadly speaking, bots use the href attribute in an <a> tag to discover pages. If you use JavaScript to make non-link elements clickable (like <button onclick=”…”>), bots will not trigger the click event and will fail to discover the destination page. If you must use a button, ensure a regular <a href> hyperlink also exists, or better yet, use CSS to style a traditional link tag to look like a button. For optimal discovery, links to pages should look this in the HTML code:
<a href="/my-page">Link to a page</a>
  • Structural Location: Google’s Quality Rater Guidelines differentiate between “main content” and “supplementary content” (navigation, footers, sidebars). Links placed in supplementary content are often given lower priority compared to links within the main body content.

What are orphan pages and how do you fix them?

Orphan pages are pages that have no internal links pointing to them, effectively cutting them off from the rest of the site. There are also “near-orphan” pages with very few internal links.

While some pages should be orphaned (like ad landing pages or post-order confirmation pages), accidental orphans are highly problematic. Without internal links, bots will not discover the page, and the page will fail to rank. Even if a bot discovers it via an XML sitemap or external link, a lack of internal links signals to Google that the page is “not very critical,” causing it to carry less weight and it likely won’t perform as well in search rankings or AI citations. Orphaned pages also harm user engagement and conversions, as human visitors cannot easily navigate to these pages.

How to find orphan pages

You must compare a full list of every page on your website with a list of all crawled internal links.

  1. Find all pages: Export a list from your CMS (like WordPress using the Export All URLs plugin) or via FTP.
  2. Find all linked pages: Use a crawler tool like JetOctopus or Screaming Frog to pull all pages connected by internal links.
    1. Quick tip: Make sure the crawl is set to respect the robots.txt directives so that you are evaluating the same links Googlebot will see.
  3. Compare: Compare the lists of all pages against the list of pages with internal links. This can be done in Excel or your favorite AI tool.

    Note: Most crawlers have a built-in “Is Orphaned” filter. While this is helpful, this will only flag the URLs that the crawl tool could have known about. You still want to compare against the full list of URLs on your website to identify all orphan pages.

How to fix orphan pages

Fixing orphan pages (or near orphan pages) is one of the best ways to improve discoverability. Do not simply dump links to orphaned pages into your website’s main navigation or footer, as this creates usability nightmares and Google may ignore or discount footer links anyway. Instead, fix orphaned pages by finding highly relevant areas within the main content of other pages to place your internal links. If you review an orphaned page and truly cannot find a relevant place to link to it from your existing content, it is likely best to delete the page.

What tools monitor discovery?

To ensure Google, Bing, OpenAI and other AI services are finding everything you want them to find, you must measure three specific metrics: the total pages on your website, the number of pages Google crawls, and the number of pages Google indexes.

  • Log Files (Crawling): The most accurate, reliable way to determine if bots have crawled a URL is to review your server’s log files or use a tool like Known Agents. This shows precisely when and how often bots request specific pages and is the only way to know if AI bots are discovering your website’s pages. Pages crawled frequently are great discovery sources to place links on to other pages.
  • Google Analytics 4 (GA4): To verify if discovered pages are actually performing, check your Acquisition report (filtered by Organic Search) to see how many unique pages are receiving traffic.
  • Google Search Console – Crawl Stats: In the “By purpose” table of the Crawl Stats report, Google shows exactly how many crawls were made to discover new content versus refreshing existing content. Discovery should be higher if you have recently added new articles.
Google crawls by purpose - Refresh or Discovery
  • Google Search Console – Page Indexing Report: Compare your total website pages against the number of indexed pages shown on Google’s Page Indexing report. If you have 3,800 total pages but Google has only indexed 2,300, Google is likely missing some of your content. Conversely, if 4,200 pages are indexed, Google may be discovering low-quality, dynamically generated junk pages that shouldn’t be indexed.
  • Google Search Console – URL Inspection: If you manually inspect a URL in Google Search Console and see “URL is unknown to Google,” it confirms discovery has completely failed. You can request indexing here to encourage crawling.

Key Takeaways: Discovery

  • Add multiple, relevant standard HTML links (<a href>) to pages within prominent main content to maximize discovery.
  • Avoid using Javascript-based routing or <button onclick> tags for important internal links.
  • Use XML sitemaps to support standard internal links, ensuring they contain accurate, live URLs. For more complex websites, you can also use HTML sitemaps.
  • Monitor your site metrics (total pages vs. crawled vs. indexed) using GSC, evaluate what pages get crawled in log files, and what pages get traffic in GA4.
  • Routinely audit for orphan pages and fix them by adding relevant main-content links or deleting the unnecessary pages entirely.

More on Modern Bot Behavior

This article is part of my series on how AI search is changing technical SEO and breaks down how we optimize for the six modern bot behaviors.

Read the next part: Crawling – how bots retrieve content from your site

Need help understanding how AI bots are using your website or want to optimize for AI search? Contact me and let’s talk.