Statistical Significance and Type I & Type II Error

Statistical Significance

Imagine I told you I have a magic quarter that comes up tails more often that it comes up heads.
At what point would you start to believe me?
This is the question that significance testing (or null hypothesis significance testing (NHST)) is designed to answer.
When is something far enough away from normal to say something is going on?
Null and Alternative Hypotheses
The null hypothesis is the starting assumption for significance testing - that nothing is going on, the manipulation didn't work, there is no difference between groups. Symbolized by H0
The alternative hypothesis is the opposite - something is going on, the manipulation worked, there is a difference between groups. Symbolized by H1 or Ha
Example: You are conducting a randomized controlled trial to determine whether a new medication for depression reduces self-reported symptoms of depression. You have two groups - an experimental group that received the new drug, and a control group that received a placebo.
- H0 - There is no difference between the groups, the new medication does not work better than placebo
- H1 - There is a difference between the groups - the new medication does work better than placebo.
The Normal Distribution

In null hypothesis significance testing (NHST), we are asking whether the value we found in our sample is far enough away from the expected value to count as something "going on".
The established value is p < .05 - there is a .05 chance that we will make an incorrect decision and say something is happening when really nothing is happening. If the p-value that results from our inferential test is lower than .05, we reject the null hypothesis and have support for the alternative hypothesis
A NHST can be 1-tailed or 2-tailed. If the test is 2-tailed, the .05 is divided up equally on both sides of the null hypothesis distribution. If the test is 1-tailed, the whole .05 is assigned to one side.

Type I & Type II Error
Type I Error - study concludes there is a difference between groups or conditions where a difference does not actually exist
Type II Error - study concludes there is no difference between groups or conditions where a difference does actually exist
Sensitivity or Correct Inference - the ability to detect a difference that actually exists
Specificity or Correct Rejection- the ability to conclude a difference doesn't exist when it doesn't actually exist
When we set our alpha level (e.g., p < .05), we are making a decision about the probability of a Type II error we are willing to accept.