Analyzing Competitor UX

After my MozCon presentation yesterday, I was asked the question of how you measure user experience on a competitor’s site. It is a site you don’t control so you can’t stick analytics code on it and rarely are competitors going to open up their analytics to you to review.

It is a great question to figure out. I wasn’t totally prepared to answer it on stage. So, I stumbled out two recommendations as an answer not entirely sure what was I saying would make too much sense. Judging by the reaction and Twitter responses, I apparently did make some sense. I wanted to elaborate a bit, though in this post.

The first recommendation is to do usability testing on your competitor’s site—either in person, remote, a five-second test, or something similar. This is a great way to get some ideas about what people are thinking about your competitor’s site and areas they struggle with. The downside is that usability testing (especially in person) can be tough to get approval or budget for—it is hard enough to get approval to do usability testing on your own site. (Though, if you are doing usability testing on your site, be sure to also look at competitor sites too as part of the testing process.)

But another approach to take, and the one that can be a bit more helpful, involves copying your competitor’s site to a test area you control then measuring the results. The approach basically breaks down as follows:

  1. Copy the page(s) on your competitor’s site that you want to test out to a testing or staging environment. Two tips:
    • Keep things easy and just keep all the pages you are testing static. Saving a page in the browser and use Save Page As -> Web Page Complete so you can get the HTML and associated files. Then upload the files for each page. You will probably need to tweak a few links to string different pages together. With this approach, you can have a test environment setup in no time.
    • Be nice and ethical. Make sure you noindex, nofollow this page or just outright block it by requiring a password to access this page. You do not want your copy of the competitor’s page to be indexed or accessible outside of testing (one, because this would help your competitor and two, it keeps lawyers way happier).
  2. Setup all the analytics of things you’d want to know about the competitor’s UX on the test environment version of the pages. Although my general approach is to keep the analytics simple, in this instance, go crazy and setup everything you can possibly think of to track.
  3. Using a service like Microworkers, pay for traffic to come to your testing environment. Do your best to target the right kind of traffic, at a minimum by country. Give the test traffic general guidelines but avoid anything terribly specific. For example, don’t ask people to make a purchase on an ecommerce site, but ask them to consider purchasing this product if they are interested. This more closely mirrors what the competitor’s real traffic might look like (and likely what your traffic might look like).
  4. Once the data is collected, tear down the test site and get to analyzing. You’ll start to understand where people might be struggling to interact on a competitor’s site. What links are they motivated to link? Where do they scroll? Where do they move their mouse? And more.

Obviously, you are never going to know exactly what is happening on your competitor’s site unless a competitor grants you access to their metrics. But, using techniques like this is a great way to figure out what may or may not be working. From there, you can get some ideas on how to change things on your site and some great ideas for things to split test.

Twitter Responses