What are WAF rules? Why do they matter for SEO and GEO? What do we need to do about WAF rules?
What are WAF rules?
WAF stands for Web Application Firewall. It is a program that is designed to monitor all the traffic coming to your website. You can define rules that tell this program to filter or block certain types of visits to a website. Some types of visitors can also be challenged. If you’ve ever been asked to verify you are a human before visiting a website, that’s a WAF rule.
WAF rules are mainly used to prevent malicious visitors from accessing the website. This protects the website from various types of exploits and attacks. For example, you might have a rule that says:
IF visitor looks automated AND visitor is not on the verified/approved list THEN challenge or block the requestVerification is one of the best ways of protecting from malicious attacks. With a service like Cloudflare, you can verify that a visitor claiming to be Googlebot actually is from Google. That way, a spam bot cannot get into your website by pretending to be a different type of bot.
Why do WAF rules matter for SEO or GEO?
WAF rules have gotten more attention in the last couple of years because you can use WAF rules to block certain bots from accessing your website. The rule might be set as:
IF crawler is classified as an AI training bot THEN block accessThis is what a lot of companies are currently doing for different types of AI bots, especially AI training bots. Blocking AI bots could negatively impact a website’s ability to appear in AI responses.
The problem is you might be blocking AI training bots without knowing it. Starting September 15, Cloudflare adjusted its default rules to block AI training bots on pages with ads (their post):
“Bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy ‘Block AI bots’ option.”
The catch in that rule is that Cloudflare’s “Mixed-purpose crawlers” includes Googlebot and Bingbot because these bots can be used for both search indexing and AI training. Cloudflare has added a separate “Disallow AI Training” option designed to avoid this specific problem. However, it is easy to make a mistake in these configurations. What this really gets to is the danger of using broad categories instead of targeting specific user agents. That leads to…
What do SEOs Need to Do About WAF Rules?
It is important to review how your WAF rules and related bot protections are defined and make sure they align with your expectations.
For a lot of our clients we’re working with, the marketing team and security team don’t always align on which bots and agents should access the website. A security team might create a perfectly reasonable rule that challenges automated traffic in an effort to protect the website, only to inadvertently challenge Googlebot, Bingbot, AI search crawlers, or AI agents. To be fair, those of us on the marketing side may be more willing to let too much automated traffic in. Both sides have a point.
So, SEOs and marketers need to talk with the security team. Ask about:
- Does the website use a WAF or bot-management product, like Cloudflare?
- What rules are currently applied to various types of automated visitors?
- Are those rules applied to broad bot categories or specific bots?
- How do existing rules align with the bots and agents that should access the site?
- How often were those rules applied? How much traffic was blocked and what specific traffic was blocked?
- What happens when a bot is blocked? Does it receive a 403, 429, challenge page or something else?
- Do WAF rules and other bot policies align with robots.txt files? Allowing in robots.txt doesn’t matter if a WAF rule blocks it.
Also, when reviewing, remember to test multiple page types, not just the homepage. You’ll also want to check static resources. I’ve seen instances where bots can access the HTML but not the JavaScript and CSS required to actually see the page.
Bottom line: SEO/GEO is not about configuring WAF rules and related bot policies. That is best left to security professionals who understand those details and understand the risks of malicious bot or agent access. However, it is up to the SEO/GEO team to define which automated visitors need access and verify that they actually have it.