Putting Data Into Perspective

By Matthew Edgar · Last Updated: January 23, 2018

The biggest struggle with data is that it is ridiculously easy to lie and manipulate with it, even if you don’t mean to do so. Sometimes this is done with malice as you bend the data to make your case. Often, though, this happens out of laziness and not taking the time to really consider the statements being made about the data.

For example, let’s consider these statements:

• Our traffic is up 42% month over month.
• We had a 12% increase in orders since releasing our new product line.
• The CTR increased 14% following these changes.

All of these statements refer to a specific data point. All attempt to show perspective by comparing metrics (traffic, orders, CTR) across time ranges. All of these statements seem actionable: Our traffic is up? Great, keep investing in those ads/content marketing/social media/whatever! Orders are up – people love that new product line – let’s invest more! The changes we made did a great job increasing CTR – let’s make more changes like it!

Except, the problem is none of these claims put the data into the right context. As a result, all of these statements tell you a compelling and attractive lie. That lie will easily mislead you into making the wrong decisions. Let’s go one by one:

Our traffic is up 42% month over month. This statement compares month over month, but what about year over year? Maybe this last month is just normally a busy month? Also, this statement doesn’t say anything about the reason the traffic up, so deciding to continue an investment in a marketing effort based solely on this is weak…maybe the traffic spiked in a channel you didn’t actively invest in? Finally, this statement tells you nothing about the traffic quality – did that 42% jump result in 42% more conversions (whether that conversion is viewing ads or placing orders)?

We had a 12% increase in orders since releasing our new product line. Did you catch how manipulative this one is? It implies a causal relationship between the release of the new product line and the increase in orders. But it doesn’t actually say the new product line is the cause. Sure, orders went up 12% since the release, but was the entirety of that increase the new product line? Or was that increase in an older product line? As well, if that increase is from the new product line, what happened to the old product line – did it go up or down?

The CTR increased 14% following these changes. Here again, is this just a coincidence or is there a causal relationship between the changes and the increase in the click through rate? Were these changes split tested–if so, what did the control group show? Even if you can’t run a split test given lower traffic volume, you can do a time comparison with a control group of other similar pages, posts, or ads–how did that control group fare? Either way, it is also important to put these changes to a repeated test to see if you get the same increase on a second (third, fourth, fifth, etc.) run. Until you have that, it is hard to say if this increase is worth pursuing more deeply.

In light of those questions, the responses to these statements could change dramatically. Instead of investing in the same marketing channels, maybe you should invest elsewhere. Instead of that new product line, maybe an older product line deserves a fresh look. And maybe those changes that once improved the CTR don’t hold up on a re-test, in which case other changes should be considered.

Ultimately, don’t ever take a statement claiming some improvement, decline, or stagnation at face value even if it sounds like it might be complete, compelling, and actionable. Take a moment to pull the statement apart a bit and see it from a different angle. The more you demand to know about the data and the more you demand to fully understand the context, the more likely you are to make the proper decision on how to move forward.