Technical SEO and AI search are full of jargon, acronyms, and technical phrases. I have assembled this glossary to help make sense of all these words and phrases. Each term in this glossary includes a simple definition to help you get a baseline understanding of the terms and their meaning, as well as links to additional information. Learning these terms and having a better understanding of what they mean will help improve your technical SEO work.

Crawling

  • Crawling: The process of bots fetching, or attempting to fetch, content and other information from URLs. Learn more about how bots crawl.
  • Bots: A complex collection of different programs that search engines and AI systems use to understand and evaluate websites.
  • Crawl Budget: The volume of hits and requests bots make to your site. Managing crawl budget effectively means ensuring bots can efficiently discover, fetch, and render your most important pages without overloading server resources.
  • Robots.txt: A plain text file located in the website’s root directory. The robots.txt file contains directives indicating how bots should crawl the website.
  • Disallow: A rule in a text file that specifically instructs robots not to crawl a certain URL or directory, though the URL might still be discovered and indexed via external links. Note that the disallow is more of a suggestion that bots do not have to follow.
  • Googlebot: Google’s automated programs (robots, spiders, or crawlers) that discover, fetch, and render content from websites. (Bing’s equivalent is Bingbot.)
  • ChatGPT-User: OpenAI’s user interaction bot. This bot is sent out in response to a user’s prompt to fetch content from a website for possible inclusion in the prompt response.
  • GPTBot: OpenAI’s training bot. This bot discovers and fetches content from websites that can be used to train OpenAI’s Large Language Model (LLM).

More details: Learn more about the different SEO and GEO bots visiting your website.

Discovery

  • Discovery: This is the initial process by which automated services (search engine crawlers, AI bots, and AI agents) identify new URLs or content for the very first time, primarily through internal and external links. 
  • Log Files: A server record of every file accessed on a website by humans or bots, including what pages were requested, when, the response status code, and the user agent.
  • Internal Link: A link from one page of a website to another page on that same website. There are numerous problems with internal links that can impact SEO and GEO performance.
  • External Link: A link from one website to another website.
  • Orphan Pages: Indexable pages that have zero internal links pointing to them. These pages are difficult for bots to discover and signal that the page is likely unimportant. (Near-orphan pages are pages with almost no internal links pointing to them.)
  • XML Sitemap: XML sitemaps are XML files that provide search engines and AI bots with information about the pages on your website, helping them identify existing URLs and establish the canonical versions of those pages.

More details: Learn more about how search and AI bots discover your website’s content.

Rendering & Page Structure

  • Rendering: The process where a browser or bot executes code (especially JavaScript and CSS) to fully load the page so all visual and structural content can be extracted.
  • Client-Side Rendering (CSR): When the server sends back an incomplete HTML shell containing JavaScript, requiring the visitor’s browser (or the bot) to execute that JavaScript to fully build and load the page.
  • Server-Side Rendering (SSR): The process where the web server fully processes the HTML code before sending it to the visitor’s browser or a crawling bot, making content immediately accessible without relying on JavaScript execution.
  • Prerendering: Rendering content in advance using an engine that executes JavaScript and delivers a static HTML snapshot to bots so they do not have to process complex scripts.
  • DOM (Document Object Model): A hierarchical, tree-like representation of a website’s HTML structure created after the browser or bot processes the initial code.
  • Accessibility Tree: A subset of the DOM that shows only the elements meaningful for navigation or interaction (like headings, links, and text areas). AI agents increasingly use this to understand how to interact with a page.
  • Semantic HTML: HTML tags that describe the meaning and structural hierarchy of content (like <nav>, <article>, or <main>), rather than just relying on generic tags (like <div> or <span>). Using semantic HTML aids accessibility and AI agent understanding.

More details: Learn more about the rendering process and how it affects SEO.

Indexing & Content Evaluation

  • Indexing: The process of storing crawled and rendered content in a structured way (such as categorizing it in a database) so it can be retrieved during search queries.
  • Noindex: An explicit directive telling bots not to index a page or rank it in search results. The bot must be able to crawl the page to see this directive. Learn more about using noindex tags.
  • Nofollow (Meta Robots Nofollow): A meta robots nofollow instructs robots not to crawl any of the links on that specific page.
  • Passage-Level Indexing: Indexing chunks of content (like individual sentences or paragraphs) from a page rather than just the whole page, allowing specific, highly relevant sections to rank or be cited by AI even if the broader page covers multiple topics.
  • Canonical URL: The official or preferred version of a page’s URL defined by a canonical tag, which helps search engines resolve duplicate content issues when a page can be accessed via multiple URLs.
  • Duplicate Content: Two or more pages on a website that return identical or nearly identical content, or serve an identical purpose. Duplicate content can confuse bots and dilute rankings.
  • Main Content (MC): Any content on the page that directly helps the page achieve its core purpose.
  • Supplementary Content (SC): Content that supports the main content but does not relate to the page’s direct purpose (e.g., navigation menus, sidebars, or footers).
  • Rel Nofollow / Link Qualifiers: Attributes (like rel=”nofollow”, rel=”sponsored”, or rel=”ugc”) added to specific links to explain the relationship of the outbound link and indicate that the link does not imply endorsement.
  • E-E-A-T: Stands for Experience, Expertise, Authoritativeness, and Trust. These are the criteria Google uses to evaluate the quality and credibility of content creators and websites.
  • YMYL (Your Money or Your Life): Topics that can significantly impact a person’s health, financial stability, safety, or society’s well-being, which Google holds to stricter E-E-A-T standards.

More details: Learn more about how indexing works for SEO and AI Search.

Website Speed & Performance (Metrics)

  • Core Web Vitals (CWV). Core Web Vitals is part of the Google Page Experience Algorithm and is designed to measure three aspects of a website’s user experience related to speed: Interaction to Next Paint (INP), Largest Contentful Paint (LCP), and Cumulative Layout Shift (CLS). 
  • TTFB (Time to First Byte): Measures how long it takes from when a URL is requested to when the first byte of information is returned from the server.
  • FCP (First Contentful Paint): Measures the time it takes for the first visible element (text, image, or SVG) to appear on the page.
  • LCP (Largest Contentful Paint): A Core Web Vitals metric that measures how long a visitor must wait before the largest visible element (like a hero image or main heading) fully renders within the viewport.
  • CLS (Cumulative Layout Shift): A Core Web Vitals metric that measures visual stability by tracking how much page elements move around unexpectedly while the page is loading.
  • INP (Interaction to Next Paint): A Core Web Vitals metric that tracks responsiveness by measuring the time between a user interaction (mouse click, screen tap, or key press) and the first visual update resulting from it.
  • Transfer Size: The amount of compressed data that must be transmitted from a web server to a browser or robot to load a web page.
  • Compression: Shrinking file sizes (typically using Gzip or Brotli algorithms) by identifying repetitive data or reducing details, minimizing the data sent from the server to the browser.
  • Minification: Removing unnecessary, unused, and/or redundant code (especially JavaScript and CSS code) without affecting the page’s functionality or design.

More details: Learn more about Core Web Vitals and the other key metrics to use to evaluate your website’s speed.

  • Generative AI Search: An answer engine powered by a Large Language Model (LLM) designed to create a comprehensive, conversational response to a complex prompt, rather than merely returning a list of ranked links.
  • LLM (Large Language Model): AI systems trained on massive amounts of textual data to learn language patterns, structures, and semantic relationships so they can statistically predict words and generate responses.
  • Vector Embeddings: A numerical format (a list of numbers mapped in multidimensional space) representing the meaning and relationships of words, phrases, or even whole documents. Embeddings are utilized by LLMs to match queries to concepts without relying on exact keywords.
  • RAG (Retrieval-Augmented Generation): A system that retrieves real-time, relevant documents from a knowledge base to supplement the LLM’s training, providing context for a more accurate, up-to-date AI response.
  • Training Bots: AI crawlers (like GPTBot) focused on gathering vast amounts of web content to build the underlying statistical language models.
  • User Interaction Bots: Bots (like ChatGPT-User) that visit and interact with websites in real-time on behalf of a specific user prompt to fetch or summarize requested information.
  • AI Agents: Goal-oriented programs designed to autonomously perform complex tasks (such as researching products or filling out forms) by planning, acting, and observing outcomes on a website.
  • Topic Modeling: AI-driven techniques (like BERTopic) used to find patterns and themes in unstructured website content by grouping semantically related words together, helping map a site as a comprehensive knowledge base.
  • Agent Cards: Small machine-readable JSON files (e.g., agent.json) using the Agent-to-Agent (A2A) Protocol that detail an AI agent’s capabilities, skills, and authentication rules so different agents can collaborate.
  • WebMCP: A new standard, WebMCP (Web Model Context Protocol) gives websites a way to support AI agents by adding declarative attributes directly to HTML elements (like forms). This explicitly defines actions for AI agents, making agentic interactions more reliable.

More details: Learn more about optimizing your website for AI agents and how AI search changes technical SEO.

Schema & Structured Data

  • Schema Markup (Structured Data): Schema is code added to the HTML that structures information, providing machine-readable details to search engines and AI robots so they can better understand content and present enhanced search results (rich results).
  • JSON-LD: JavaScript Object Notation for Linked Data. This is a format of plain text for structuring data that is highly readable by both search engines and AI bots. It does not require JavaScript execution to parse.
  • Microdata: A specific format used to write schema markup that integrates structured, machine-readable data directly into your webpage’s existing HTML. Both Google and Bing support Microdata, however it is not as robust or expressive as JSON-LD. (RDFa is similar to Microdata and adds schema markup as part of inline HTML.)

More details: Learn more about schema and how to utilize schema on your website.

Redirects

  • Redirect: A redirect sends visitors from one URL to another. The URL redirected from is called the redirect source or origin. The URL redirected to is referred to as the redirect destination or target.
  • Redirect Chain: A redirect chain is where a URL redirects multiple times before arriving at the destination. Each redirect in the chain is called a hop. Robots waste resources crawling through redirect chains and may simply stop following the chain after a certain number of hops.
  • Redirect Loop: A redirect chain that circles back onto itself is called a redirect loop. This results in an error because no destination can be reached. 
  • 301 / 308 Redirect: HTTP response codes for redirects that indicate a permanent move. Redirects with these status codes are more likely to pass SEO value from the old URL to the new destination.
  • 302 / 307 Redirect: HTTP response codes that indicate a temporary URL change. Redirects with temporary status codes may be less likely to pass SEO value.

More details: Learn more about how to use redirects.

Technical Directives & HTTP Status Codes

  • HTTP Headers: Text fields sent between the browser/robot and server that contain metadata about a request (like User-Agent) or the returned file (like caching or encoding directives).
  • X-Robots-Tag: An HTTP header that operates like the meta robots tag. A noindex or nofollow can be specified. This is typically used on images or PDFs where the HTML meta tag cannot be used.
  • User Agent: An HTTP Header specifying the name of the browser, bot or agent requesting the page. The server can be programmed to respond differently based on the User Agent.
  • HTTP Response Status Code: A numerical code that indicates the page’s status, indicating if the page is operating correctly, is in error, requires authentication, and more.
  • 200 OK: The standard HTTP response status code indicating that the page loaded successfully and everything is normal.
  • 404 Not Found: A response code indicating the requested file or page could not be located on the server.
  • 410 Gone: A response code explicitly indicating that a requested file was purposefully and permanently removed from the website.
  • Soft 404: A misconfiguration where a page’s content indicates the requested file is not available, but the server erroneously returns a 200 “OK” status code.
  • 5xx Server Errors (e.g., 500, 503): Response codes indicating a server-side failure or overload, preventing the browser or robot from accessing the content.

More details: Learn more about HTTP Status Codes and how to test the status codes on your website.

Other Important SEO Terms

  • Algorithm (Algo, Algo Updates): A set of instructions programmed into a computer to solve a particular problem. Search engines have a variety of algorithms used to fetch, transform, evaluate, and rank data from websites. Google regularly updates its algorithms with algorithm updates (or algo updates). 
  • Search Engine Results Page (SERP): The page seen after conducting a search lists a variety of results for the search. This is referred to as the search engine results page or SERP. The SERP contains a variety of search result listings. Those listings include web pages, images, and various features added by the search engine.
  • Rank (Ranking Position): After crawling and indexing a website, search engines will review the information extracted from each website contained in the index to determine where the website ought to appear, or rank, in search results (if it should appear at all). If and when a website ranks, it ranks in a particular position on the SERP.
  • Click (SERP Click): The number of people who clicked to a page on the website from the search engine result page (or SERP). This includes clicks on various search features but does not include clicks on ads. 
  • Impression (SERP Impression): The number of times people saw a website listed on search results, not including any ads. 
  • Title Tag: An HTML tag that contains the main name of the page. This isn’t displayed to visitors on the page itself, but is available in the code for bots and agents to access.
  • Meta Tag: An HTML tag that contains additional information about the webpage. This can include a description summarizing the page’s content (meta description), how robots are to interact with the page (meta robots), and more.