High School
SAT
ACT
University
MCAT
LSAT
DAT
Wize University Statistics Textbook > Multiple Regression

Multiple Regression Model

Prediction Intervals for YPrevious Section
Hypothesis Testing for Multiple RegressionNext Section
Popular Courses
Find My Course
0:00 / 0:00

Multiple Regression



So far, for simple linear regression, we only used one explanatory variable xx to explain or predict one response variable yy. In reality, it may take more than one explanatory variable to explain yy.

Suppose we use the number of hours spent studying (xx) to predict one’s grade (yy) and we get this simple linear regression equation:

y^=48.56+3.599x\hat{y}=48.56+3.599x
Given this linear regression model, r2=0.671r^2=0.671. This means only 67% of grade is explained by the number of hours spent studying.


PAGE BREAK

Where is the other 33%?

It is explained by other explanatory variables other than hours spent studying, such as IQ, GPA, and hours spent playing video games.

When more explanatory variables are added to a simple regression model to strengthen the ability to explain yy, the simple regression model is converted into a multiple regression model.

PAGE BREAK
When we have multiple explanatory variables to explain y, we are using a multiple regression model:

Y=βo+β1X1+β2X2+......+βkXk+ε\boxed{Y=\beta_o+\beta_1X_1+\beta_2X_2+......+\beta_kX_k+\varepsilon}
k=k= # of explanatory variables,xix_i^{ }; i=1,2,3,...,ki=1,2,3,...,k

ε=\varepsilon= random error term. It represents everything that the model does not explain for y.

The estimated regression equation is:

y^=bo+b1x1+.....+bkxk\displaystyle\boxed{{\hat{y}=b_o+b_1x_1+.....+b_kx_k}}

To predict grade using a multiple regression line, it may look like this:


y^=bo+b1(Study)+b2(IQ)+b3(GPA)+b4(Games)\hat{y}=b_o+b_1\left(Study\right)+b_2\left(IQ\right)+b_3\left(GPA\right)+b_4\left(Games\right)


PAGE BREAK

Simple Linear Regression vs. Multiple Regression

Simple Linear Regression
  • k=1k=1: There is one explanatory variable XX to explain YY
  • there is one t-score
  • correlation rr can be used
  • R2R^2 may be used

Multiple Regression
  • k>1k>1: There are multiple explanatory variables X1,X2,...XnX_1,X_2,...X_n to explain YY
  • each explanatory variable has their own t-score
  • correlation rr cannot be used
  • R2R^2 may be used
  • F-score




0:00 / 0:00

F-test for Multiple Regression

What the "F"?

You have probably noticed the F-score in the ANOVA table. What is it? In regression, it is important to know the difference between a t-test and an F-test.
  • The F-stat only tells us if the overall model is sufficient. However, it does not tell us which individual explanatory variables are significant.
  • Each explanatory variable will have its own t-score so you will be able to assess the significance of each one by running t-tests.
  • There is only one F-score in a regression model.


0:00 / 0:00

Solving for the F-score

We use the F-score to test to see if the regression model improves our ability to predict yy. In other words, we test the overall significance of the regression model.


Ho:H_o: β1=β2=β3=...=βk=0\beta_1=\beta_2=\beta_3=...=\beta_k=0 ("The overall model is not significant.”)
Ha:H_a: at least one βi≠0\ \beta_i\ne0 ("The overall model is significant.”)

"F" for "full" model
F=SSRkSSE(n−k−1)=MSRMSE\displaystyle\boxed{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}}

where,
  • k=k= # of explanatory variables, xi′sx_i{'}s

The F-test has two degrees of freedom:

  • df numerator=k=k
  • df denominator=n−k−1=n-k-1

PAGE BREAK

Summary:



PAGE BREAK
Example

Determine the F-statistic:
F=SSRkSSE(n−k−1)=MSRMSE\displaystyle{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}}


F=951.841467.66(8−1−1)=951.8477.94=12.21\displaystyle{F=\frac{\frac{951.84}{1}}{\frac{467.66}{\left(8-1-1\right)}}=\frac{951.84}{77.94}=12.21}



PAGE BREAK

The follow rules are only true in simple linear regression where is there is only one explanatory variable (k=1)(k=1):

  • p-value(F) = p-value(t)
0.0129 = 0.0129

  • t2=Ft^2=F
(3.4946)2=12.21\left(3.4946\right)^2=12.21


Watch Out!
The above rules are only true if k=1k=1. This is not the case in multiple regression where k>1k>1.




0:00 / 0:00

Example: Solving for the F-score

Solve for F by completing the table.




k+(n−k−1)=n−1k+\left(n-k-1\right)=n-1
k=1k=1
n−1=12n-1=12

Therefore:
1+(n−k−1)=121+\left(n-k-1\right)=12
n−k−1=12−1=11n-k-1=12-1=11

Don't overthink it:
(Regression df) + (Residual df) = Total
1+11 = 12

SST=SSR+SSESST=SSR+SSE
49519=SSR+3308549519=SSR+33085
SSR=16434SSR=16434

MSR=SSRk=164341=16434\displaystyle{MSR=\frac{SSR}{k}=\frac{16434}{1}=16434}

MSE=SSEn−k−1=3308513−1−1=3308511=3007.727\displaystyle{MSE=\frac{SSE}{n-k-1}=\frac{33085}{13-1-1}=\frac{33085}{11}=3007.727}

F=SSRkSSE(n−k−1)=MSRMSE=164343007.727=5.463926\displaystyle{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}=\frac{16434}{3007.727}=5.463926}


Solve for F using the clues provided.



R=0.8R=0.8
Se=12.362S_e=12.362
k=3k=3
n=60n=60

Try this on your own before looking at the video solution! Click on "Hint" for useful formulas.