Hypothesis tests
Moving the decision threshold splits the risk between the two possible errors, but does not reduce it. The only thing that lowers both at once is measuring more, and which of the two hurts more is an engineering decision.
01How a test is set up
A hypothesis test decides between two statements using data. The structure is always the same and is deliberately asymmetric: you start by assuming that nothing is going on and you only abandon that assumption if the data make it very hard to believe.
| What it claims | Its role | |
|---|---|---|
| \( H_0 \), null | There is no effect, no difference | Upheld unless proven otherwise |
| \( H_1 \), alternative | There is an effect | It is what has to be demonstrated |
The asymmetry is intentional and resembles that of a trial: innocence is not proven, it is presumed. That is why the outcome of a test is never “\( H_0 \) is true,” but “there is not enough evidence to reject it”, which is quite a bit less.
02The two errors
Any decision made with limited data can go wrong in two ways, and each has its own name because they are not interchangeable.
| \( H_0 \) is true | \( H_0 \) is false | |
|---|---|---|
| Reject | Type I error, probability \( \alpha \) | Correct: power \( 1-\beta \) |
| Do not reject | Correct | Type II error, probability \( \beta \) |
In quality control they have names of their own: the producer’s risk is rejecting a batch that was good, and the consumer’s risk is accepting one that was bad. Which of the two hurts more depends on what is being manufactured, and that decision is one of engineering, not of statistics.
The two distributions —that of \( H_0 \) and that of \( H_1 \)— with the decision threshold in between. Move it and watch how one error grows when the other drops. Then increase the sample size and see what happens to both at once.
03The p-value, and what it is not
The p-value is the probability of observing a result at least as extreme as the one obtained assuming that \( H_0 \) is true. If it is smaller than \( \alpha \), you reject.
It is not the probability that \( H_0 \) is true: it is calculated assuming that it is. It is not the probability of having made a mistake in rejecting. And it does not measure the size of the effect: with a large enough sample, a negligible difference gives a tiny p. A result can be statistically significant and practically irrelevant, and that distinction is the most important one in the whole unit.
| Situation | Decision | What it really means |
|---|---|---|
| \( p = 0.001 \), effect of 0.01 % | \( H_0 \) is rejected | The difference exists and does not matter |
| \( p = 0.20 \), effect of 30 % | Not rejected | There may be a large effect and data are lacking |
| \( p = 0.049 \) versus \( p = 0.051 \) | Reject / do not reject | They are practically the same result |
The third row is against the magic threshold. The 0.05 is a convention of Fisher’s, not a natural law, and treating 0.049 and 0.051 as opposite conclusions is an artifact. The reasonable approach is to report the p-value together with the effect size and its confidence interval, and let the reader judge.
04In the lab
Measure twenty resistors from each of two different batches of the same nominal value. State the hypothesis that the means are equal, calculate the t statistic and the p-value, and decide with \( \alpha = 0.05 \).
With the measured standard deviation, calculate what difference between batches could be detected with 80 % power using twenty measurements per batch. Recalculate for fifty and compare.
Simulate two sets of a thousand data points that differ by 0.1 % of the mean. Verify that the p-value comes out very small and discuss whether that difference would justify rejecting a batch in production.
05Common mistakes
- Saying that \( H_0 \) was “accepted.” It was not rejected, which is different: the evidence may be lacking.
- Interpreting the p-value as the probability that \( H_0 \) is true. It is calculated assuming that it is.
- Confusing significant with important. With a large sample, any difference comes out significant.
- Lowering \( \alpha \) without looking at \( \beta \). Reducing one error enlarges the other.
- Choosing the hypothesis after seeing the data. That invalidates the whole test.
- Testing many things and reporting the one that worked. With twenty tests at 5 %, one comes out significant by pure chance.
06Self-assessment
What is a Type I error?
Rejecting \( H_0 \) when it is true. Its probability is \( \alpha \), the significance level.
What is the power of a test?
\( 1-\beta \): the probability of detecting an effect that really exists.
How do you reduce both errors at once?
By increasing the sample size. Moving the threshold only trades one for the other.
What does \( p = 0.03 \) mean?
That if \( H_0 \) were true, a result this extreme would occur 3 % of the time.
Can a result be significant and not matter?
Yes, and it is common. With large samples, negligible differences give very small p-values.
What is the consumer’s risk?
Accepting a batch that was bad: a Type II error in the context of quality control.
07Further reading
- Douglas C. Montgomery and George C. Runger. Applied Statistics and Probability for Engineers. 7th ed. (Spanish edition, “Probabilidad y estadística aplicadas a la ingeniería”), Wiley, 2018. Chapters 9 and 10: one- and two-sample tests, power and sample size.
- Ronald L. Wasserstein and Nicole A. Lazar. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, vol. 70, 2016. The official statement of the American Statistical Association on what a p-value does and does not say. Six pages worth reading.
- Douglas C. Montgomery. Introduction to Statistical Quality Control. 6th ed. (Spanish edition, “Introducción al control estadístico de la calidad”), Wiley, 2009. Producer’s and consumer’s risk, sampling plans and control charts applied to production.