Testing Changes On Your Website
Last Updated: December 02, 2016
As you work on improving your website, you will likely have many good ideas about what types of changes could increase traffic, get more conversions or get more people to interact with your website. Those ideas can be based on what you are seeing work when reviewing website analytics. Other times these ideas for what to change will be based on ideas you saw somewhere else. Still other times those ideas might be based on an internal debate about different ways to write your website’s copy or design a call to action.
The only way to figure out which of these changes will work, and which ones won’t, is to test it out. You can review numbers, browse other websites, and debate endlessly about which changes may or may not work. There is no shortage of ideas or opinions about those ideas. But at some point, you need to make changes and see which ideas get more traffic, get more people interacting with your website and which changes get more people converting.
As you test the changes out, some will work and some won’t. Either way presents an opportunity to learn something new about your website and the people who visit. Obviously, the changes that get you some type of an increase in traffic, interactions or conversions are the changes to keep. But both the changes that work and the ones that don’t should be studied deeply. Why did it work or why did it fail? Did it succeed or fail in a large way, or just a small improvement? How will you adjust future changes given what you’ve learned from this success or failure? For successful changes, what similar types of changes could you make?
If you have enough traffic, the best way to run these tests and compare different ideas is with a split test. In a split test, some percent of your visitors see the current, unchanged version of your website and another percent see the changed version. The biggest problem with split tests is that many people go too small and are hoping for tiny improvements. Instead, you want to test out changes that have the potential to lead to a big impact—otherwise, the results will be too small to be worth the time, money and effort invested. There are various models you can use that can help you determine which types of changes will have the better chance of success.
The other problem with split tests, though, is that your website might have too little traffic to make a split test worth the effort. If your website’s conversion rate is 1.5% currently, but you want it closer to the 2%, that requires a fairly aggressive increase of 33% from the split test. To have a statistically significant result, you’d need at least 9,200 visitors to see each version of the split test (run your own calculations). For larger websites, getting 9,200 visitors to each version may be no problem and would require only a few days or a week to run the test. For a website with 1,000 or fewer visitors per month, the split test would have to drag on for months or maybe even a full year, making it hardly worth the time invested.
That means split tests won’t work for every website. Instead, an alternative to the split test is the time comparison test. With a time comparison test, you make the changes to your website, let those changes run for a shorter period—a few weeks to a month—then see what impact those changes had on your website’s conversion rate. In order to make time comparisons work, you need to have an idea of what a normal week or month looks like for you website so that you know what difference the changes made during the week or month your website had those changes up.
For example, your average conversion rate over the per month might be 2.1% but it varies +/- 10%, so a month where the conversion rate is 1.89% or a month where the conversion rate is 2.3% is not uncommon. However, if you make changes to your website and in the month those changes are up on your website, your conversion rate might jump to 2.7%. If you see that 29% increase, you would know it is unique given the monthly average conversion rates. This suggests the changes you have made are working and you should keep them around longer.
Regardless if you use a split test or the time comparison method to see the impact of changes, you must be constantly testing and trying new things out on your website. The biggest reason websites fail is because they don’t make enough changes—your competitors, technology in general, and your visitor’s expectations are all constantly changing. You and your website have to keep pace, and that means you have to keep testing changes to see what will work (and won’t) so that you get more traffic, engagement and conversions.