Our websites are more important than ever. Our websites contain the best repository of information available for generative AI bots to learn about our companies but also our industry. In many respects, we are not just competing for rankings, mentions and citations. Rather, we are now competing to be the knowledge base of choice for your industry.
One way I’ve found to help understand our website as a knowledge base is with topic modeling. Topic modeling reveals the patterns in our website’s content by revealing themes and topis that are discussed. It does this by grouping words that frequently occur together by semantic meaning. In particular, I have found this to be helpful to compare different sites: which site has the most topics? which site covers those topics more completely?
There are many topic modeling algorithms we can use to help with this. The most common that you may be familiar with is Latent Dirichlet Allocation (LDA). Two of the better ones now available are BERTopic and Top2Vec. There is a great study (older, but still interesting) comparing how these topic modeling algorithms.
A few key points about each algorithm:
- LDA (Latent Dirichlet Allocation) relies on a probabilistic (Bayesian) algorithm. To work correctly, it tends to require more domain knowledge and fine-tuning.
- Top2Dev uses word embeddings, which are numerical representations that capture the word meaning. This is ideal for topic discovery in large, unstructured text datasets (like websites!).
- BERTopic uses transformers, meaning that along with embedding there is a greater focus on relevance. The results may be more interpretable than Top2Vec but it may also highlight more outlier topics.
These models clearly aren’t the same thing as LLMs. However, Top2Vec and BERTopic share some principles with LLMs (like embeddings and context-aware learning) even though they are simpler tools that focus on clustering and topic discovery. Still, though, we’ve found these models can be useful tools to understand how well our websites may work for generative AI and SEO.