Training: How bots learn from content

Training refers to how AI systems learn from web content, either through foundational model training or ongoing retrieval for Retrieval Augmented Generation (RAG) systems.

SEO is no longer just for Googlebot. This post is part of a series on adapting technical SEO for AI bots and new search behaviors. Explore the full series.

How does content end up in LLM training sets?

LLMs are trained on a mixture of data sources. Some of those sources are proprietary, like data feeds from partners and licensing agreements. Some of those sources are publicly available, including website content. 

To access publicly available website content, bots from AI systems crawl the web, similar to search engine robots, and fetch content. Similar to search engines, the AI system processes the content found on websites to prepare it for training. This includes assessing the content quality and removing certain types of content that the AI systems do not want included in the model. For example, OpenAI filters out spam and personal information.

Where an LLM differs from traditional search systems, though, is what they do with the content after the processing is complete. Traditional search engines attempt to organize the information contained in the processed content. In contrast, AI systems are trained to detect patterns within the processed content. The training is not about getting the AI system to learn facts or information. Instead, training is about the model learning how language is structured and how words are related. Training intends to improve how the LLM predicts words in response to a prompt. For example, during training, a model is shown incomplete sentences and must make predictions on how to finish those sentences. 

Not all content is equally valuable for training. AI systems need to train LLMs on a variety of content about a given subject so that the models can learn the different ways people speak about this topic and the different types of words that might be used. If your website’s content is short, shallow, and leaves many questions unanswered, it likely will not be used by the AI companies to train their LLMs. Instead, your website’s content should provide more depth and discuss topics from different perspectives. There is some speculation that providing content in different formats is also useful. For example, a blog post, a Reddit forum, and a podcast will all discuss the same topic in different ways. Those differences give the LLM more data to learn from, which allows the LLM to make better predictions when asked about that topic. 

What pre-processing steps are applied before training?

Before any web content is used in LLM training, it is cleaned and standardized. This is referred to as pre-processing. It matters because the content the model uses in training is not the same content seen by visitors. The content used during training is a stripped-down, text-only version of our website’s content. During pre-processing, some of our content may be removed or altered to the point that it is no longer useful. If that happens, the content will not be included in the training data. 

Pre-processing includes several steps, including tokenization (where words and phrases are broken into smaller chunks called tokens), stemming and lemmatization (where words are reduced to their root form), normalization (standardizing word usage), removing duplication, and more. Two of the more important steps to consider from a technical and structural optimization perspective are HTML stripping and boilerplate content removal.

HTML Stripping

One of the basic steps of pre-processing is HTML stripping. This is exactly what it sounds like: the content is stripped out of the HTML. We need to make sure we optimize our website for “clean text” extraction. This starts with ensuring our primary content is in plain, crawlable HTML. Content loaded by JavaScript, content in PDFs, or text inside images is harder to extract and may not be included in the training materials. Content inside more complicated HTML may also be more challenging to extract.

Along with technical considerations, there are also structural considerations. Stripping out the content will remove and flatten formatting, so clarity in plain text becomes crucial. Use consistent headers and sentence structures so that the article makes sense even with formatting removed. One simple technique I use to test this is to load a website without CSS. This can be done with the Chrome Web Developer extension. Once this is installed in Chrome, click the gear icon to open the Web Developer panel, select the CSS tab and then select “Disable All Styles”.

You can see an example of an article on my website without CSS.

Boilerplate Content Removal

Another area where pre-processing mistakes can occur is with boilerplate content removal. A good portion of a website’s design is reused across the entire website, including headers, footers, sidebars, and navigation elements. This is referred to as boilerplate content. In their Quality Rater Guidelines, Google refers to this as Supporting Content.

While boilerplate content is helpful for human visitors on our website, it is unnecessary information for bots during training. AI systems only want LLMs trained on the main content of the page because only the main content contains relevant information with useful language patterns.

Unfortunately, mistakes can happen when removing boilerplate content. A system may be overly aggressive and accidentally remove some key parts of the main content. This is likely to happen on websites that are poorly coded or within content that is poorly structured. So, it is important to test if the boilerplate content on your website will be removed correctly. An older but still helpful tool for removing boilerplate content is jusText. The following screenshot shows an example of Elementive’s website passed through a test. The boilerplate content is shown in red and the main content in green.

Should our content be part of the training material?

One of the most fundamental questions companies need to address is whether their website’s content should be included in training data. 

The upside is that by including our website in the training material, our content may influence what LLMs learn about our company, products, services, and industry. During training, LLMs are learning about language patterns and structures in order to make more accurate predictions when responding to a prompt. Website content can discuss our products or services in different mediums (blogs, videos, forums, customer testimonials, etc.) and from different perspectives (i.e. what your customers say vs. what your company’s experts say). This provides the LLM with helpful information that can be used to train it about how to discuss what your company does with potential customers. As well, by allowing LLMs to train on your website’s content, you potentially increase the chances of LLMs associating your company’s name with specific customer needs. That could result in the LLM mentioning your company more in responses than a competitor.

The downside is that by including our website in training material, the LLM may learn how to discuss what our company does so well that potential customers have little need to come to our website. This is especially true for informational websites. Informational websites already provide a variety of content, often covering a subject in depth. Informational websites often provide this content in different formats, including text and video. All of that makes the content on an informational website highly valuable to LLMs during the training process. Most informational websites, though, receive no compensation from the AI companies behind the LLMs. In these cases, inclusion in training materials may not make sense.

My recommendation is to evaluate how your customers are using LLMs and determine if LLMs are more competitive (learning so much from your website that people no longer need your company) or if LLMs are assistive (LLMs learn how to discuss your website and help explain the benefits of working with your company). 

Whether you opt to have your website included in LLM training data, remember that the LLMs will still be trained on some information about your company. Your company is discussed elsewhere on the web outside of your website. Your company might be mentioned in social media posts, forums, news articles, reviews, directory listings, and more. At least some of those mentions will be picked up by LLMs and used to influence how these models discuss your company. Even if your website is not included in training materials, remember to check external sources.

How is content used in RAG pipelines?

When a user submits a prompt to an AI system, the main component of that system responding to the user is the LLM. By itself, an LLM is capable of generating a full response to the prompt. The LLM’s response will be based on a series of predictions about which words to use and predictions about how to structure those words. A well-trained LLM is capable of constructing a helpful, useful response to prompts. However, an LLM is trained to be a pattern-recognition system. It is not trained to provide accurate information. This is why LLM responses contain inaccurate information and hallucinations.

To address this, most AI systems are now using a Retrieval Augmented Generation (RAG) system to supplement LLM responses. The RAG is not pre-trained like the LLM and is not intended to be a pattern-recognition system. Instead, it operates almost like a search engine. The RAG fetches content similar to the user’s prompt from a knowledge base and passes that content to the LLM. The LLM brings this retrieved content alongside the user’s prompt when generating a response. This provides the LLM with more context so that it can, typically, provide a more accurate response.

The most important consideration SEOs need to keep in mind is that this means there are two systems we need to optimize for in AI search: the knowledge base and the LLM. This differs from traditional search, where optimization is aimed at a single system (the index). By ensuring our content is used in the training data and that our content is stored in the knowledge base, we increase the likelihood that our company is referenced or even favored when a response is generated. 

Side note: There are two key things to keep in mind about how RAG systems work alongside LLMs. 

• First, the knowledge base may contain some of the same information used in the LLM’s training materials. However, the knowledge base often includes different, especially newer information, than what was in the training materials. LLMs are not updated that often, but a knowledge base can be updated more frequently. 
• Second, the RAG system uses vector search to query the knowledge base. A vector search looks for similarity of meaning, intent, context, and relationships. I discussed this in more detail in the following video.

Should I use LLMS.txt files to help bots learn about site content?

LLMS.txt is a proposed standard meant to give AI systems an easier way to find and interpret the content on your site. In some respects, this is a special kind of sitemap that is written in Markdown. With an LLMS.txt file, you can list out some or all pages on the website, add short descriptions, and provide context about each file. An AI tool might use this information when deciding how to reference your content.

While the idea sounds promising, there are challenges. For one, no major AI provider has endorsed LLMS.txt. Google has already said they do not and will not support it, and OpenAI and Perplexity have not made any commitments. Right now, the only indication of adoption is from Anthropic (Claude).

Even if support grows, there are risks. LLMS.txt could be easily manipulated, since site owners can describe their content however they like, without verification. That makes it unclear how much trust an AI system would place in these files. There’s also redundancy because this file duplicates what’s already available through well-structured HTML and XML sitemaps.

All that said, it might be worth testing this file to see if it changes how AI bots are learning from your website’s content. If you want to experiment, keep it small and safe:

• Test with a limited set of pages, not your whole site. Track whether those pages see more AI bot activity or mentions.
• Add an X-Robots noindex header so the file itself doesn’t appear in search results.
• Avoid manipulative summaries and spam. Keep it accurate and fair.

The bigger takeaway is that LLMS.txt is not a shortcut to speed up training. If bots are struggling to learn from your site, the better fix is usually to improve the fundamentals about how bots crawl and render the website. That includes using clear HTML, semantic headings, strong internal linking, and well-structured content. Those improvements help all bots, human readers, and AI systems alike.

Key Takeaways: Training

• LLMs are trained to recognize language patterns, not memorize facts. To be considered valuable for training, your website’s content must be detailed, well-structured, and written from diverse perspectives that help models understand a topic from multiple angles.
• If your content is hidden behind JavaScript, buried in images or PDFs, or structured with overly complex HTML, it may be stripped out during preprocessing. Clean, crawlable HTML and strong content hierarchy help ensure your content survives the preprocessing steps and can be used effectively by AI systems.
• AI systems strip out repeated page elements like headers, footers, and navigation. If your main content is embedded in a confusing or repetitive structure, you risk having important content mistakenly removed during preprocessing.
• Unlike LLMs, which are pre-trained, RAG systems actively retrieve content from a knowledge base in response to a user’s prompt in real-time. To increase your chances of being cited in AI responses, you need content that is not only included when training the LLM but also stored in the RAG system’s knowledge base.

More on Modern Bot Behavior

This article is part of my series on how AI search is changing technical SEO and breaks down how we optimize for the six modern bot behaviors.

Read the next part: Interaction: How bots engage with your site on behalf of users

Need help understanding how AI bots are using your website or want to optimize for AI search? Contact me and let’s talk.