Multiple Regression Model

Multiple Regression

So far, for simple linear regression, we only used one explanatory variable x to explain or predict one response variable y. In reality, it may take more than one explanatory variable to explain y.
Suppose we use the number of hours spent studying (x) to predict one’s grade (y) and we get this simple linear regression equation:
y^=48.56+3.599x
Given this linear regression model, r2=0.671. This means only 67% of grade is explained by the number of hours spent studying.
Where is the other 33%?
It is explained by other explanatory variables other than hours spent studying, such as IQ, GPA, and hours spent playing video games.
When more explanatory variables are added to a simple regression model to strengthen the ability to explain y, the simple regression model is converted into a multiple regression model.
When we have multiple explanatory variables to explain y, we are using a multiple regression model:
Y=βo+β1X1+β2X2+......+βkXk+ε
k= # of explanatory variables,xi; i=1,2,3,...,k
ε= random error term. It represents everything that the model does not explain for y.
The estimated regression equation is:
y^=bo+b1x1+.....+bkxk
To predict grade using a multiple regression line, it may look like this:
y^=bo+b1(Study)+b2(IQ)+b3(GPA)+b4(Games)
Simple Linear Regression vs. Multiple Regression
Simple Linear Regression
- k=1: There is one explanatory variable X to explain Y
- there is one t-score
- correlation r can be used
- R2 may be used
Multiple Regression
- k>1: There are multiple explanatory variables X1,X2,...Xn to explain Y
- each explanatory variable has their own t-score
- correlation r cannot be used
- R2 may be used
- F-score


F-test for Multiple Regression
What the "F"?
You have probably noticed the F-score in the ANOVA table. What is it? In regression, it is important to know the difference between a t-test and an F-test.
- The F-stat only tells us if the overall model is sufficient. However, it does not tell us which individual explanatory variables are significant.
- Each explanatory variable will have its own t-score so you will be able to assess the significance of each one by running t-tests.
- There is only one F-score in a regression model.


Solving for the F-score
We use the F-score to test to see if the regression model improves our ability to predict y. In other words, we test the overall significance of the regression model.
Ho: β1=β2=β3=...=βk=0 ("The overall model is not significant.”)
Ha: at least one βi=0 ("The overall model is significant.”)
"F" for "full" model
where,
- k= # of explanatory variables, xi′s
The F-test has two degrees of freedom:
- df numerator=k
- df denominator=n−k−1
Summary:

Example

Determine the F-statistic:
F=(8−1−1)467.661951.84=77.94951.84=12.21
The follow rules are only true in simple linear regression where is there is only one explanatory variable (k=1):
- p-value(F) = p-value(t)
0.0129 = 0.0129
- t2=F
(3.4946)2=12.21

Example: Solving for the F-score
Solve for F by completing the table.

k+(n−k−1)=n−1
k=1
n−1=12
Therefore:
1+(n−k−1)=12
n−k−1=12−1=11
Don't overthink it:
(Regression df) + (Residual df) = Total
1+11 = 12
SST=SSR+SSE
49519=SSR+33085
SSR=16434
MSR=kSSR=116434=16434
MSE=n−k−1SSE=13−1−133085=1133085=3007.727