Wize University Statistics Textbook > Simple Linear Regression

Predictions and Residual Plots

Predictions

When you have the linear regression equation $\hat{y}=b_0+b_1x$ , you can make predictions along it.

The variable $x_i$ predicts $\hat{y_i}$ . In other words, you plug in $x_i$ to solve for $\hat y_i\$ .

Example

Portions of information contained in this publication/book are printed with permission of Minitab, LLC. All such material remains the exclusive property and copyright of Minitab, LLC. All rights reserved.

Example

\hat{y}=58.18+2.52x

If Phil studied for 12 hours, what is his predicted grade?

$\hat{y}=58.18+2.52(12)=88.42$

Brendan studied for 20 hours. Can we predict he’ll get 109%?

No - we cannot extrapolate! We can only make predictions within the data range $x=0$ to $15$

Residuals (Errors)

A residual (or "error") is the vertical distance between the observed value (or actual value) $y_i$ and the predicted value $\hat{y_i}$ for all values of $x_i$ in the data set.

$\boxed{e_i=y_i-\hat{y_i}}$

Example

\hat{y}=58.18+2.52x

What is the residual for $x=12$ ?

$\hat{y}=58.18+2.52(12)\approx88$

From the table in the previous page, the observed value for $x=12$ is $y_i=85$ .

Since we predicted $\hat{y_i}$ = 88, the residual for $x=12$ is:

$e_i=y_i-\hat{y_i}$

$e_i=85-88=-3$

The equation $\hat{y}=35+1.50x$ is used to predict the hourly wage $y$ of someone with $x$ years of experience.

Elizabeth has 4 years of experience and earns $36 per hour. What is the residual?

−$5

$41

Not enough information.

I don't know

Residual Plots

A residual plot ( $e_i$ on y-axis and $x_i$ on x-axis) helps us assess if the regression model is appropriate or not. It is appropriate if the residual plot shows no pattern.

In the residual plot (on the right), where the residuals $e_i=y_i-\hat{y_i}$ are on the y-axis and plotted against the x-variable for every data point ( $x_i,y_i$ ) .
This residual plot in particular shows a nice linear relationship because the residuals fall randomly above and below the “0” line.

Standard Deviation of the Residuals

The standard error of the residuals is a measure of the accuracy of predictions. The smaller the standard error of the estimate is, the more accurate the predictions are.

S_e=\sqrt{\frac{SSE}{n-2}}=\sqrt{\frac{SS_{yy}-\frac{(SS_{xy})^2}{SS_{xx}}}{n-2}}

The reason n − 2 is used (and not n − 1) is that two parameters (i.e. the slope and the intercept) were estimated in order to estimate the sum of squares.

Regression Diagnostics (L.I.N.E.)

A linear regression model may not always be the most appropriate for the data you have. You should assess the appropriateness of the model by examining your residual plot by checking if any of the conditions are violated.

Common Types of Violated Conditions

Match the residual plots with the conditions violated.

L. Linearity condition violated.

I. Independence condition violated.

N. Normality condition violated (skewed)

E. Equal spread (constant variance) condition violated.

Image illustrating Regression Diagnostics

Top row (left to right): L, N

Bottom row (left to right): E, I

Here is a residual plot for selling price (Y) vs. size of a condo in square-feet (X).

Illustration for practice question: Simple Linear Regression

Which condition is violated?

Linearity

Independence

Normality

Equal Spread

I don't know

Standard Deviation of the Residuals

The standard error of the residuals (or standard error of the estimate) is a measure of the accuracy of predictions. Specifically, it measures how far the data points are from the best fit regression line overall. In regression, it is also loosely called the overall standard deviation.

The smaller the standard error of the estimate is, the more accurate the predictions are overall.

Linear regression where $k=1$ (one explanatory variable):

\displaystyle\boxed{S_e=\sqrt{\frac{SS_{yy}-\frac{(SS_{xy})^2}{SS_{xx}}}{n-2}}}

General formula for any value of $k$ :

\displaystyle\boxed{S_e=\sqrt{\frac{SS_{yy}-\frac{(SS_{xy})^2}{SS_{xx}}}{n-k-1}}}

Standard Deviation of the Residuals vs. R Squared

In addition to R Squared, the standard deviation of the residuals is used to access goodness-of-fit in regression analysis.

There is a negative correlation between $R^2$ and $S_e$ :

As $S_e$ decreases, $R^2$ increases.
A low $S_e$ tells us that the distances between the data points and the fitted values are small overall, indicating better fit; hence a higher $R^2$ .

As $S_e$ increases, $R^2$ decreases.
A high $S_e$ tells us that the distances between the data points and the fitted values are large overall, indicating poorer fit; hence a lower $R^2$ .

Examples

Lower $\colorThree{S_e}$

Image illustrating Standard Deviation of the Residuals

Since the overall standard deviation is relatively low, the can make more accurate predictions.
Residuals will be low for predictions.

Higher $\colorThree{S_e}$

Since the overall standard deviation is relatively high, the predictions will not be as accurate.
Residuals will be higher for predictions.

Practice: Standard Deviation of the Residuals

Using the information below, solve for the standard deviation of the residuals $S_e$ .

$n=5\\SS_{xx}=50\\SS_{yy}=518\\SS_{xy}=159$

$k=1$ (simple linear regression)

2.03

4.13

12.38

13.10

I don't know

Extra Practice

Simple Linear Regression

Here is a residual plot for selling price (Y) vs. size of a condo in square-feet (X).

Which condition is violated?

Predictions and Residual Plots

The equation $\hat{y}$ =35+1.50 $x$ is used to predict the hourly wage $y$ of someone with $x$ years of experience.

Residual Plots

Here are three residual plots. Which of the following is true?

Correlation and Residual Plots

Pamela and Tim are trying to be healthy by taking the stairs instead of the elevator. We want to see there is a relationship between the floor you live on and the time it takes to get to your floor from the lobby. Pamela lives on the 8th floor. Tim lives in the penthouse on the 20th floor.

$\sum_{ }^{ }x^2=258$

$\sum_{ }^{ }y^2=687$

Regression line: Predictions and Residual Plots

True or false?

If the sum of the residuals is equal to 0, then the best-fitting regression line goes through all the data points.

Predictions: Residual Plots

(v) Mike actually has 8% body fat. What is the residual?

Predictions and Residual Plots: Regression Diagnostics Question

Let’s take a look at our residual plot in our example:

Regression Diagnostics (check all that applies):

[ ] Outliers

Predictions and Residual Plots

The regression model $y=4000-2.7x$ is used to estimate the annual number of cars going over the speed limit (y) based on the number of police officers on the road. Estimate the number of cars going over the speed limit when there are 40 police officers on the road.

Predictions and Residual Plots

You wish to predict the number of times someone logs in their email a day. The estimated linear regression model is $Logins\ =\ 5\ +\ 1.3\left(work\right)$ where Logins is the number of times a person logs in their email a week and work is the number of hours of work done a week. Logan works 30 hours a week and checks his email 49 times a week. What is his residual?

Predictions and Residual Plots

One day George wanted to examine the linear relationship between how many hours a student studies ( $x$ ) the day before the exam and the exam grade ( $y$ ).

He asks 13 random students and collects the following data

$r=0.819\ \ s_x=3.24\ \ s_y=14.24\ \ \overline{x}=8.25\ \ \overline{y}=78.25$

Residual Plots: Calculation

In investigating the relationship between number of people on a person's phone list ( $x$ ) and the number of text messages per day ( $y$ ), 40 individuals were surveyed and the following information was recorded:

\sum_{i=1}^{40}x_i=700 \quad s_x=3.5

\sum_{i=1}^{40}y_i=3200 \quad s_y=18.9 \quad r=0.67

Predictions: Residual Plots

\sum_{i=1}^{40}x_i=700\quad\ \ \ s_x=3.5

\sum_{i=1}^{40}y_i=3200\ \ \quad s_y=18.9\quad\ \ r=0.67

Predictions and residual plots

The following assertions about the three residual are True or False

Residual Plot A: No problem – unbiased and homoscedastic (constant variance)

Residual Plot B: Heteroscedasticity – we see a “fan”, indication of non-constant variance across the range of values

Predictions and Residual Plots

Cindy calculated residuals and find that all of them are negative. Which of the following is true?

Predictions and Residual Plots

\sum_{i=1}^{40}x_i=700 \quad s_x=3.5

\sum_{i=1}^{40}y_i=3200 \quad s_y=18.9 \quad r=0.67

Predictions and Residual Plots

Suppose we want to see if there is a relationship between the size of a condo unit and the selling price of it. We randomly sampled 16 units, here are the results and the regression output:

(a) Estimate the linear regression model.

(b) Unit #3 is 1250 sqft, what is his predicted selling price? Compute the residual for this estimate.

Predictions and Residual Plots

Suppose we want to see if there is a relationship between the size of a condo unit and the selling price of it. We randomly sampled 16 units:

(a) Estimate the linear regression model.

(b) Unit #3 is 1250 sqft, what is his predicted selling price? Compute the residual for this estimate.

Predictions and Residual Plots

You wish to predict the number of times someone logs in their email a day.

The estimated linear regression model is $Logins=5+1.3\left(work\right)$

where Logins is the number of times a person logs in their email a week and

Predictions and Residual Plots

In linear regression, which of the following indicates that we should fit the scatter plot with a line of best fit?

Predictions: Residual Plots

\sum_{i=1}^{40}x_i=700 \quad s_x=3.5

\sum_{i=1}^{40}y_i=3200 \quad s_y=18.9 \quad r=0.67

Linear Regression: Residual Plots

The equation $\hat{y}=35+1.50x$ is used to predict the hourly wage $y$ of someone with $x$ years of experience.

Elizabeth has 4 years of experience and earns $36 per hour. What is the residual?

Predictions and Residual Plots

Suppose the following summary data was given:

x-variable: $\mu_x=10,\ s_x=2$
y-variable: $\mu_y=250,\ s_y=10$

Predictions and Residual Plots

Create a free account to get started

Predictions

Create a free account to get started

Residuals (Errors)

Create a free account to get started

Create a free account to get started

Residual Plots

Standard Deviation of the Residuals

Create a free account to get started

Regression Diagnostics (L.I.N.E.)

Common Types of Violated Conditions

Create a free account to get started

Create a free account to get started

Standard Deviation of the Residuals

Practice: Standard Deviation of the Residuals

Create a free account to get started

Extra Practice