The importance of statistical tests – statistical significance and confidence interval

When you start doing more serious research, you will need to conduct a statistical test. It helps you to determine whether your hypothesis is significant or can be rejected. If you start browsing for statistical analyses online, you will find a large number of possible test that can be chosen depending on your task. In this post, let us focus on a simple example of statistical analysis that would allow us to understand what is statistical significance and what is confidence interval.

Building a test scenario

To understand the matter lets use some random data which is distributed normally. For this, we can use 100 casual observations of the price tag for the new Nokia 3310 from different sources. The average price is around 50€ with a standard deviation of 10€

Now lets plot how those samples are distributed along with the standard distribution graph:

As you can see the probability of distribution is Gaussian with a mean of 50 and standard deviation 10. For later intuition we will be using this Gaussian distribution expressed as probability density calculated by the formula:

We already know the mean and standard deviation from our data, so calculation and plotting become easy.

Now we can get back to the statistical test.

For the sake of simplicity, let us say after one a year you decide to recheck the prices of Nokia 3310 and find that its average dropped to 23€.

Can we state that price is different? From first glance, it looks very different, but what if the collected data sample is different? What if you receive the same size of the data sample and you get an average price closer to 50€? The statistical test helps to answer a question whether the price is significantly different from previous – in other words, is sample mean significantly different from the null hypothesis. So how to draw a line where we can state that result is significantly different?

Significance level

To have some limits we choose some significance level α which indicates the risk that the difference exists even if there is no difference. Usually, this level is selected as α = 0.05. This means that there is a 5% probability of rejecting the null hypothesis. Humanly speaking if we think that the average price of Nokia has dropped after a year, there is still a 5% probability that it’s not.

After we get significance level, we can draw those limits on distribution graph. On distribution graph Y axis is representing probability distribution, we need to calculate the area under the chart to find actual probability limits. Since the diagram is double sided, we need to split the 0.05 significance level to both sides having 0.025 for each side. Total area under the graph is equal to 1. In zoomed graph portion you can see that significance level is at a price around 22€:

Because price our observed price is 23€ doesn’t fall in this critical region, so we must state that price didn’t change. So we cannot reject the null hypothesis of an average price being 50€. In other words, our observed average price of 23€ isn’t statistically significant to reject the null hypothesis. Significance level can be different, for instance, if we choose significance level α = 0.1, then definitely our observed new price will fall into the risky area, and so we could state that price changed significantly.

Probability value P

It seems that we have covered the basics, but there is still one more important thing to discuss – the P value. Practically speaking P represents the probability of obtaining observed data (in our case price is 23€). So we need to find an area under the distribution graph up to point 23€. One side is 0.0023. Add both sides we get 0.052 which is the value of P = 0.052.

Now we can rephrase our investigation. Comparing null hypothesis of Nokia price being 50€ to the recent observation of average price 23€ we find that probability P=0.052 of the cost is lower than significance level α = 0.05 and so we state that we stick to null hypothesis and say that price isn’t lowered that 50€. I hope this makes sense.

Final thoughts about the statistical significance

As mentioned before, the significance level isn’t a fixed value. Before you test your hypothesis, you must choose your level. The common ones are 0.05 and 0.01. You choose significance level with the thought of error rate that you reject the null hypothesis even if it is true. So P value and significance level help you to quantify the results of observation and control the error in the hypothesis test.

Sometimes instead of significance level, e.g., 0.05 you will find a notation of confidence interval equal 95% which is practically the same if you subtract 0.05 from 1 you get 0.95.

Confidence interval indicates uncertainty interval around null hypothesis point (50€) or in other words margin error.

You can play around with data above with this [Matlab live editor file.zip]

Leave a Reply