Decision Errors
The likelihood of making a type 1 error and likelihood of making a type 2 error are actually inversely proportional. So, it's actually not that easy to keep both of those error rates down. So sometimes we have to choose and we're going to talk about how do we choose between which one we are okay with being a little higher versus which one we really want to minimize as much as possible.
For example, Declaring the defendant guilty when they are actually innocent (type 1) is much worse than declaring the defendant innocent when they are actually guilty (type 2).
Type 1 error
We reject H0 when the p-value is less than 0.05 (alpha = 0.05). This means that, for those cases where H0 is actually true, we do not want to incorrectly reject it more than 5% of those times. In other words, when using a 5% significance level there is about 5% chance of making a Type 1 error if the null hypothesis is true.
This is why we prefer small values of alpha – increasing alpha increases the Type 1 error rate.
Type 2 error
If the alternative hypothesis is actually true, what is the chance that we make a Type 2 error, i.e. we fail to reject the null hypothesis even when we should reject it?
- The answer is not obvious.
- If the true population average is very close to the null value, it will be difficult to detect a difference (and reject H0).
- If the true population average is very different from the null value, it will be easier to detect a difference.
- Clearly, Beta depends on the effect size (the difference between point estimate and null value)
