High School
SAT
ACT
University
MCAT
LSAT
DAT
Wize University Psychology Textbook > Research Methods in Psychology

Statistical Significance and Type I & Type II Error

Distributional ThinkingPrevious Section
Descriptive Statistics Next Section
Popular Courses
Find My Course
0:00 / 0:00

Statistical Significance



Imagine I told you I have a magic quarter that comes up tails more often that it comes up heads.

At what point would you start to believe me?

This is the question that significance testing (or null hypothesis significance testing (NHST)) is designed to answer.

When is something far enough away from normal to say something is going on?





Photo by Watchduck / CC BY

PAGE BREAK

Null and Alternative Hypotheses

The null hypothesis is the starting assumption for significance testing - that nothing is going on, the manipulation didn't work, there is no difference between groups. Symbolized by H0

The alternative hypothesis is the opposite - something is going on, the manipulation worked, there is a difference between groups. Symbolized by H1 or Ha

Example: You are conducting a randomized controlled trial to determine whether a new medication for depression reduces self-reported symptoms of depression. You have two groups - an experimental group that received the new drug, and a control group that received a placebo.
  • H0 - There is no difference between the groups, the new medication does not work better than placebo
  • H1 - There is a difference between the groups - the new medication does work better than placebo.

PAGE BREAK

The Normal Distribution

Photo by Smahdavi4 / CC BY

In null hypothesis significance testing (NHST), we are asking whether the value we found in our sample is far enough away from the expected value to count as something "going on".

The established value is p < .05 - there is a .05 chance that we will make an incorrect decision and say something is happening when really nothing is happening. If the p-value that results from our inferential test is lower than .05, we reject the null hypothesis and have support for the alternative hypothesis


Wize Tip
Unless a passage or question explicitly gives you a different p-value as the "critical value" or "test value", assume it is p < .05

PAGE BREAK

A NHST can be 1-tailed or 2-tailed. If the test is 2-tailed, the .05 is divided up equally on both sides of the null hypothesis distribution. If the test is 1-tailed, the whole .05 is assigned to one side.


Watch Out!
Look for alternative hypothesis that say one condition will be better or worse than the other - this is a clue that a 1-tailed test is being performed. If you hypothesize that the results will come out one way (e.g., X is better than Y) and it comes out the other way (e.g., Y is better than X) you cannot switch sides if you've committed to a 1-tailed test!

0:00 / 0:00

Type I & Type II Error



Type I Error - study concludes there is a difference between groups or conditions where a difference does not actually exist

Type II Error - study concludes there is no difference between groups or conditions where a difference does actually exist

Sensitivity or Correct Inference - the ability to detect a difference that actually exists

Specificity or Correct Rejection- the ability to conclude a difference doesn't exist when it doesn't actually exist

When we set our alpha level (e.g., p < .05), we are making a decision about the probability of a Type II error we are willing to accept.

checklist
Mark Yourself Question
  1. Grab a piece of paper and try this problem yourself.
  2. When you're done, check the "I have answered this question" box below.
  3. View the solution and report whether you got it right or wrong.

Practice: Statistical Significance

You are conducing a study to determine whether a regimen of electrolyte replacement drinks will result in more normal heart rhythms in the elderly following mild to moderate exercise, compared to plain water.

1. What are the H0 and H1?
2. Is this a one-tailed or two-tailed hypothesis?

Practice: Type I & Type II Error

Bags of chips have a label stating that the weight of the contents is 6 oz. A consumer watchdog group thinks the bags are under‐filled and decides to test them. A Type I error in this situation would mean: