Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

Multiple Regression

Image illustrating Assessing a Multiple Regression Model



So far, for simple linear regression, we only used one explanatory variable xx to explain or predict one response variable yy. In reality, it may take more than one explanatory variable to explain yy.


Suppose we use the number of hours spent studying (xx) to predict one’s grade (yy) and we get this simple linear regression equation:


y^=48.56+3.599x\hat{y}=48.56+3.599x

Given this linear regression model, r2=0.671r^2=0.671. This means only 67% of grade is explained by the number of hours spent studying.




Where is the other 33%?


It is explained by other explanatory variables other than hours spent studying, such as IQ, GPA, and hours spent playing video games.


When more explanatory variables are added to a simple regression model to strengthen the ability to explain yy, the simple regression model is converted into a multiple regression model.


When we have multiple explanatory variables to explain y, we are using a multiple regression model:


Y=βo+β1X1+β2X2+......+βkXk+ε\boxed{Y=\beta_o+\beta_1X_1+\beta_2X_2+......+\beta_kX_k+\varepsilon}

k=k= # of explanatory variables,xix_i^{ }; i=1,2,3,...,ki=1,2,3,...,k


ε=\varepsilon= random error term. It represents everything that the model does not explain for y.


The estimated regression equation is:


y^=bo+b1x1+.....+bkxk\displaystyle\boxed{{\hat{y}=b_o+b_1x_1+.....+b_kx_k}}


To predict grade using a multiple regression line, it may look like this:



y^=bo+b1(Study)+b2(IQ)+b3(GPA)+b4(Games)\hat{y}=b_o+b_1\left(Study\right)+b_2\left(IQ\right)+b_3\left(GPA\right)+b_4\left(Games\right)



Simple Linear Regression vs. Multiple Regression

Simple Linear Regression

  • k=1k=1: There is one explanatory variable XX to explain YY
  • there is one t-score
  • correlation rr can be used
  • R2R^2 may be used


Multiple Regression

  • k>1k>1: There are multiple explanatory variables X1,X2,...XnX_1,X_2,...X_n to explain YY
  • each explanatory variable has their own t-score
  • correlation rr cannot be used
  • R2R^2 may be used
  • F-score


Image illustrating Assessing a Multiple Regression Model




Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

F-test for Multiple Regression

What the "F"?


You have probably noticed the F-score in the ANOVA table. What is it? In regression, it is important to know the difference between a t-test and an F-test.

  • The F-stat only tells us if the overall model is sufficient. However, it does not tell us which individual explanatory variables are significant.
  • Each explanatory variable will have its own t-score so you will be able to assess the significance of each one by running t-tests.
  • There is only one F-score in a regression model.

Image illustrating F-test for Multiple Regression



Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

Solving for the F-score

We use the F-score to test to see if the regression model improves our ability to predict yy. In other words, we test the overall significance of the regression model.



Ho:H_o: β1=β2=β3=...=βk=0\beta_1=\beta_2=\beta_3=...=\beta_k=0 ("The overall model is not significant.”)

Ha:H_a: at least one βi≠0\ \beta_i\ne0 ("The overall model is significant.”)


"F" for "full" model

F=SSRkSSE(n−k−1)=MSRMSE\displaystyle\boxed{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}}


where,

  • k=k= # of explanatory variables, xi′sx_i{'}s


The F-test has two degrees of freedom:


  • df numerator=k=k
  • df denominator=n−k−1=n-k-1



Summary:


Image illustrating Solving for the F-score



Example


Image illustrating Solving for the F-score

Determine the F-statistic:

F=SSRkSSE(n−k−1)=MSRMSE\displaystyle{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}}



F=951.841467.66(8−1−1)=951.8477.94=12.21\displaystyle{F=\frac{\frac{951.84}{1}}{\frac{467.66}{\left(8-1-1\right)}}=\frac{951.84}{77.94}=12.21}





The follow rules are only true in simple linear regression where is there is only one explanatory variable (k=1)(k=1):


  • p-value(F) = p-value(t)

0.0129 = 0.0129


  • t2=Ft^2=F

(3.4946)2=12.21\left(3.4946\right)^2=12.21







Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

Example: Solving for the F-score

Solve for F by completing the table.



Image illustrating Example


Image illustrating Example


k+(n−k−1)=n−1k+\left(n-k-1\right)=n-1

k=1k=1

n−1=12n-1=12


Therefore:

1+(n−k−1)=121+\left(n-k-1\right)=12

n−k−1=12−1=11n-k-1=12-1=11


Don't overthink it:

(Regression df) + (Residual df) = Total

1+11 = 12


SST=SSR+SSESST=SSR+SSE

49519=SSR+3308549519=SSR+33085

SSR=16434SSR=16434


MSR=SSRk=164341=16434\displaystyle{MSR=\frac{SSR}{k}=\frac{16434}{1}=16434}


MSE=SSEn−k−1=3308513−1−1=3308511=3007.727\displaystyle{MSE=\frac{SSE}{n-k-1}=\frac{33085}{13-1-1}=\frac{33085}{11}=3007.727}


F=SSRkSSE(n−k−1)=MSRMSE=164343007.727=5.463926\displaystyle{F=\frac{\frac{SSR}{k}}{\frac{SSE}{\left(n-k-1\right)}}=\frac{MSR}{MSE}=\frac{16434}{3007.727}=5.463926}



Solve for F using the clues provided.


Illustration for practice question: F-score



R=0.8R=0.8

Se=12.362S_e=12.362

k=3k=3

n=60n=60


Try this on your own before looking at the video solution! Click on "Hint" for useful formulas.

Your answers

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors