Review of Type I and Type II Errors

Inference is based on probability, so there is always some chance of making a wrong decision. Recall that two types of wrong decisions can be made in hypothesis testing. When we reject a null hypothesis that is true, we commit a type I error. When we fail to reject a null hypothesis that is false, we commit a type II error.

The following table summarizes the logic behind type I and type II errors.

It is possible to have some influence over the likelihoods of committing these errors, but decreasing the chance of a type I error increases the chance of a type II error. We have to decide which error is more serious for a given situation. Sometimes a type I error is more serious, and other times a type II error is more serious.

Decreasing the Chance of Type I or Type II Error

How can we decrease the chance of a type I or type II error? Because decreasing the chance of a type I error increases the chance of a type II error, we have to weigh the consequences of these errors before deciding how to proceed.

Recall that the probability of committing a type I error is α. Why? Well, when we choose a level of significance (α) , we are choosing a benchmark for rejecting the null hypothesis. If the null hypothesis is true, then the probability that we will reject a true null hypothesis is α. So the smaller α is, the smaller the probability of a type I error.

It is more complicated to calculate the probability of a type II error. The best way to reduce the probability of a type II error is to increase the sample size. But once the sample size is set, larger values of α will decrease the probability of a type II error (while increasing the probability of a type I error).

Following are general guidelines for choosing a level of significance:

• If the consequences of a type I error are more serious, choose a small level of significance (α).
• If the consequences of a type II error are more serious, choose a larger level of significance (α). But remember that the level of significance is the probability of committing a type I error.
• In general, we pick the largest level of significance that we can tolerate as the chance of a type I error.

Note: It is not always the case that one type of error is worse than the other.

Let’s Summarize

• Hypothesis tests for two proportions can answer research questions about two populations or two treatments that involve categorical data.
• The null hypothesis for the two-proportions test is always a statement of “no difference.”
  • H₀: p₁ − p₂ = 0

  The alternative hypothesis is one of the following.

  • H_a: p₁ − p₂ < 0, or
  • H_a: p₁ − p₂ > 0, or
  • H_a: p₁ − p₂ ≠ 0
• The test statistic for the two proportions test is similar to the test statistic for one sample proportion tests.

  [latex]Z\text{}=\text{}\frac{\mathrm{statistic}\text{}-\text{}\mathrm{parameter}}{\mathrm{standard}\text{}\mathrm{error}}[/latex]

  [latex]Z\text{}=\text{}\frac{(\mathrm{difference}\text{}\mathrm{in}\text{}\mathrm{sample}\text{}\mathrm{proportions})\text{}-\text{}(\mathrm{difference}\text{}\mathrm{in}\text{}\mathrm{population}\text{}\mathrm{proportions})}{\mathrm{standard}\text{}\mathrm{error}}[/latex]

  [latex]Z\text{}=\text{}\frac{({\stackrel{ˆ}{p}}_{1}-{\stackrel{ˆ}{p}}_{2})\text{}-\text{}({p}_{1}-{p}_{2})}{\sqrt{\frac{\stackrel{ˆ}{p}\text{}(1-\stackrel{ˆ}{p})\text{}}{{n}_{1}}+\frac{\stackrel{ˆ}{p}\text{}(1-\stackrel{ˆ}{p})}{{n}_{2}}}}[/latex]

  This statistic is approximately normal in its distribution if each sample has at least ten successes and failures. Note that the standard error is estimated with pooled proportion.

• The normal distribution may be used to provide P-values for a two-proportions test if each sample has at least 10 successes and failures.
• When the P-value in a two-proportions test is less than the level of significance (α), we should reject the null hypothesis in favor of the alternative. In this case, we say that the differences are statistically significant.
• Two types of errors can be made when conducting a hypothesis test.
  • A type I error occurs when we reject a true null hypothesis.
  • A type II error occurs when we fail to reject a false null hypothesis.
  • The level of significance, α, is the probability of a type I error.
  • Increasing the sample size lowers the probability of a type II error.
  • After considering the consequences of the type I and II errors, we should choose the largest value for α that we can tolerate, because increasing α decreases the probability of a type II error.
• After conducting a hypothesis test, it is important to consider whether the conclusions are reasonable. We discussed two common pitfalls in drawing conclusions from statistical studies.(1) The conclusion is not appropriate to the study design.
  • The study makes an inference based on nonrandom data.
  • The study makes inappropriate cause-and-effect conclusions.
  • The study overgeneralizes its conclusions.

  (2) The conclusion confuses statistical significance with practical importance.