Hypothesis Test for Difference in Two Population Proportions (6 of 6)

 

Learning Objectives

  • Identify type I and type II errors and select an appropriate significance level based on an analysis of the consequences of each type of error.

Review of Type I and Type II Errors

Inference is based on probability, so there is always some chance of making a wrong decision. Recall that two types of wrong decisions can be made in hypothesis testing. When we reject a null hypothesis that is true, we commit a type I error. When we fail to reject a null hypothesis that is false, we commit a type II error.

The following table summarizes the logic behind type I and type II errors.

This table summarizes the logic behind type I and type II errors. If Ho is true: We reject Ho (accept Ha) is a Type I error. We fail to reject Ho (not enough evidence to accept Ha) is a correct decision. If Ho is false (Ha is true) We reject Ho (accept Ha) is a correct decision. We fail to reject Ho (not enough evidence to accept Ha) is a Type II error.

It is possible to have some influence over the likelihoods of committing these errors, but decreasing the chance of a type I error increases the chance of a type II error. We have to decide which error is more serious for a given situation. Sometimes a type I error is more serious, and other times a type II error is more serious.

Learn By Doing

Teens and Antidepressants

Recall the description of a clinical trial in which researchers study the effect of a new antidepressant on teens. Researchers design a randomized, controlled, double-blind experiment to study the effect of the antidepressant Fluoxetine combined with psychiatric therapy. The control group receives a placebo and psychiatric therapy. The response variable is improvement, which means symptoms of depression improve.

The hypotheses are as follows, with p1 = proportion of teens who improve in the treatment group (Fluoxetine and psychiatric therapy) and p2 = proportion of teens who improve in the control group (placebo and psychiatric therapy).

  • H0: p1 − p2 = 0
  • Ha: p1 − p2 > 0

Decreasing the Chance of Type I or Type II Error

How can we decrease the chance of a type I or type II error? Because decreasing the chance of a type I error increases the chance of a type II error, we have to weigh the consequences of these errors before deciding how to proceed.

Recall that the probability of committing a type I error is α. Why? Well, when we choose a level of significance (α) , we are choosing a benchmark for rejecting the null hypothesis. If the null hypothesis is true, then the probability that we will reject a true null hypothesis is α. So the smaller α is, the smaller the probability of a type I error.

It is more complicated to calculate the probability of a type II error. The best way to reduce the probability of a type II error is to increase the sample size. But once the sample size is set, larger values of α will decrease the probability of a type II error (while increasing the probability of a type I error).

Following are general guidelines for choosing a level of significance:

  • If the consequences of a type I error are more serious, choose a small level of significance (α).
  • If the consequences of a type II error are more serious, choose a larger level of significance (α). But remember that the level of significance is the probability of committing a type I error.
  • In general, we pick the largest level of significance that we can tolerate as the chance of a type I error.

Note: It is not always the case that one type of error is worse than the other.

Learn By Doing

Hormone Replacement Therapy

Recall the experiment that investigated the side effects of hormone replacement therapy (HRT) for women with menopausal symptoms. The experiment randomly assigned over 16,000 U.S. women to receive a hormone treatment or a placebo. The experiment was double blind. After 5 years, a larger proportion of the hormone group had breast cancer and heart disease. This observed difference was statistically significant. Researchers were so alarmed by the results that the experiment was ended early to prevent further harm to the health of the women participating in the hormone group.

The type I error in this situation is that we conclude that HRT increases the risk of breast cancer and heart disease, but it does not. The type II error is that we conclude that HRT does not increase the risk of breast cancer and heart disease, but it does.

Identify the type of error associated with each consequence.

 

 

Let’s Summarize

  • Hypothesis tests for two proportions can answer research questions about two populations or two treatments that involve categorical data.
  • The null hypothesis for the two-proportions test is always a statement of “no difference.”
    • H0: p1 − p2 = 0

    The alternative hypothesis is one of the following.

    • Ha: p1 − p2 < 0, or
    • Ha: p1 − p2 > 0, or
    • Ha: p1 − p2 ≠ 0
  • The test statistic for the two proportions test is similar to the test statistic for one sample proportion tests.

    [latex]Z\text{}=\text{}\frac{\mathrm{statistic}\text{}-\text{}\mathrm{parameter}}{\mathrm{standard}\text{}\mathrm{error}}[/latex]

    [latex]Z\text{}=\text{}\frac{(\mathrm{difference}\text{}\mathrm{in}\text{}\mathrm{sample}\text{}\mathrm{proportions})\text{}-\text{}(\mathrm{difference}\text{}\mathrm{in}\text{}\mathrm{population}\text{}\mathrm{proportions})}{\mathrm{standard}\text{}\mathrm{error}}[/latex]

    [latex]Z\text{}=\text{}\frac{({\stackrel{ˆ}{p}}_{1}-{\stackrel{ˆ}{p}}_{2})\text{}-\text{}({p}_{1}-{p}_{2})}{\sqrt{\frac{\stackrel{ˆ}{p}\text{}(1-\stackrel{ˆ}{p})\text{}}{{n}_{1}}+\frac{\stackrel{ˆ}{p}\text{}(1-\stackrel{ˆ}{p})}{{n}_{2}}}}[/latex]

    This statistic is approximately normal in its distribution if each sample has at least ten successes and failures. Note that the standard error is estimated with pooled proportion.

  • The normal distribution may be used to provide P-values for a two-proportions test if each sample has at least 10 successes and failures.
  • When the P-value in a two-proportions test is less than the level of significance (α), we should reject the null hypothesis in favor of the alternative. In this case, we say that the differences are statistically significant.
  • Two types of errors can be made when conducting a hypothesis test.
    • A type I error occurs when we reject a true null hypothesis.
    • A type II error occurs when we fail to reject a false null hypothesis.
    • The level of significance, α, is the probability of a type I error.
    • Increasing the sample size lowers the probability of a type II error.
    • After considering the consequences of the type I and II errors, we should choose the largest value for α that we can tolerate, because increasing α decreases the probability of a type II error.
  • After conducting a hypothesis test, it is important to consider whether the conclusions are reasonable. We discussed two common pitfalls in drawing conclusions from statistical studies.(1) The conclusion is not appropriate to the study design.
    • The study makes an inference based on nonrandom data.
    • The study makes inappropriate cause-and-effect conclusions.
    • The study overgeneralizes its conclusions.

    (2) The conclusion confuses statistical significance with practical importance.