In part 1 of this 3 part series, our resident statistician, Caspar Yuill, helped us get our head around Normal Distribution with a little help from Marathon legend, Eliud Kipchoge. In this follow-up, Caspar takes a look at probability and how you can use it to make decisions.
Problem: Improving Customer Experience
You’re the Head of Customer Experience for a national retail chain. You’ve felt for a while that your bricks and mortar stores need a new fit-out in order to provide a better in-store experience. So, you decide to test an upgrade out on one location, then compare performance against your other locations to determine whether there’s an impact on sales or overall customer satisfaction.
At the end of the experiment, you have a couple of months of data to compare. As you’re analysing it, you find a small but noticeable difference. The contracts to refit the rest of the stores are there. But it’s an expensive project. One thought keeps niggling at you: “How sure can you be that the investment is going to pay off?”
Principle: The Vitamin Test
To answer this, let’s take a step back and look at how and why the field of statistics was invented. Statistics is used to compare one set of data to another and decide how different those two groups are. Usually, you’re comparing two groups, and determining how different the average of one group is to the average of the other.
For example, say you’re a bit of mad scientist, and you’ve designed a new vitamin blend that you think will increase height. Naturally, you want to test this out, so knowing the average height of a 15-year-old, you select a group of 15-year-olds, give them the supplement, and measure their changes in height over the time they’re taking the supplement. You compare the results of your test group against the average, and it looks like this:
The height mean of all 15 year olds versus the height of our supplement test group
It appears that the supplement did work. But what if, by pure chance, you selected 15-year-old boys that were genetically taller to begin with? Perhaps you accidentally selected all the starting players for the local high school basketball team? That could be another reason for the results you’re seeing. In that case, the supplement actually wouldn’t have been the reason for the height differences.
Such a result would be what’s called the null hypothesis.
The null hypothesis is the hypothesis that the differences between groups is purely due to random sampling, and that there are no underlying differences between them (caused by the experiment we’ve conducted, for example). Through the way the scientific method is designed, we want to disprove that assumption by the strength of our results.
When we’re using statistics, we’re trying to confirm the difference between our groups is due to the intervention we’re making, and not due to random sampling. Therefore, we’re trying to reject the null hypothesis.
Where P Values Come In
P Values, or Probability Values, represent the likelihood that the null hypothesis is correct (i.e. that your results are due to random sampling, and there’s no meaningful difference between the groups). More specifically, P Values tell you the number of times you’d randomly get the result you’re seeing if you repeated the experiment 100 times. The lower the P Value, the less likely it is you’re seeing natural variance, and the more likely you’re seeing a significant difference from your standard population.
If you get a P Value of 0.05 for your supplement experiment, that would indicate if you were to repeat the same experiment 100 times, you’d get a significant difference between groups 5 times out of a 100, simply due to random sampling. Put another way, there is a 95% chance that the changes in height you’re seeing are due to your vitamin supplement blend.
The Problem, Part Two
Back to our Customer Experience problem. The results are in. Store Two, the location you updated, has $1,087 in average sales over the time period. Your control, Store One, has averaged $1,010 in sales. Here’s the information in a spreadsheet. Can you interpret the results and be sure that the intervention made a difference?
In Part 3, we’ll go into how to get your P Value and prove whether the intervention did, and how to answer problems related to customer experience, your vitamin experiment, and customer financial data.
