How to Use Wildcards and $ in Robots.txt
Learn how the * wildcard and $ end-of-URL character work in robots.txt, along with common mistakes that can accidentally block or allow crawling.
Read How to Use Wildcards and $ in Robots.txt →Learn how the * wildcard and $ end-of-URL character work in robots.txt, along with common mistakes that can accidentally block or allow crawling.
Read How to Use Wildcards and $ in Robots.txt →Is WordPress the best CMS for SEO? Compare WordPress, Webflow, Shopify, Squarespace and AI website builders for speed, technical control and SEO.
Read What Is the Best CMS for SEO? →Working on technical SEO and/or web analytics inevitably will require getting into the code. What languages do you need to know to succeed?
Read Languages to Know for Technical SEO and AI Search →How do AI agents browse, search, and interact with our websites? What does that mean for marketing, SEO and UX?
Read What AI Agents Are Actually Doing on Websites →Learn how to correctly connect form labels and input fields to improve accessibility, help AI agents understand forms, and pass Lighthouse label tests.
Read How to Use Form Labels for Accessibility and AI Agents →https://www.youtube.
Read Is Google Fetching & Using Your Website’s XML Sitemap File? →Did GPTBot suddenly stop crawling your website? If so, don't panic! That appears to be the normal trend. Here's the data about GPTBot...
Read How does GPTBot Crawl? →https://www.youtube.
Read Has Google Indexed Your Website’s Pages? | Google Search Console Pages Report →https://www.youtube.
Read How AI Search Changes SEO – No Longer Just About Google’s Index →How much time does Google take to load your website? Slower crawls can impact performance. Here is how to find out in Google Search Console.
Read Crawl Stats: Average Response Time →https://www.youtube.
Read How to Fix 3 Common Technical SEO Problems →AI search has a measurement problem: AI bots crawl our website more often than they send visitors. How does this change measuring SEO performance?
Read Crawl-to-Refer Ratio →