The Method of Least Squares: Minimizing Squared Residuals

The method of least squares is the rule used to pick the "best" straight line through a scatter of points.


For any trial line y^=b0+b1x\hat y = b_0 + b_1x, each data point produces a residual ei=yi−y^ie_i = y_i - \hat y_i, the vertical gap between the observed value and the line. Different lines give different sets of residuals, so we need one number that scores how good a line is overall.

Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point

Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point


The least-squares criterion defines that score as the sum of squared residuals, often written SSE:

The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.

The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.


S(b0,b1)=∑i=1nei2=∑i=1n(yi−b0−b1xi)2S(b_0,b_1) = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n \big(y_i - b_0 - b_1x_i\big)^2


The "best" line, by definition, is the one that makes S(b0,b1)S(b_0,b_1) as small as possible. This discussion assumes that the model fits an intercept and that at least two observed xx-values are distinct.


Squaring, rather than adding the residuals directly, prevents positive and negative residuals from cancelling, so a line cannot look artificially perfect just because its errors balance to zero.


Under those conditions, SSE is a strictly convex quadratic in b0b_0 and b1b_1, so it has a unique minimizing pair; smoothness enables differentiation to locate it. Adding absolute residuals also prevents cancellation, but it defines a different fitting method, least absolute deviations.

Worked example

Consider the points (1,2),(2,4),(3,5)(1,2), (2,4), (3,5). Compare two candidate lines using SSE:


x

y

Line A: y^=1+1.5x\hat y=1+1.5x

Line B: y^=1.5+1x\hat y=1.5+1x

1

2

2.5 (e=-0.5, e²=0.25)

2.5 (e=-0.5, e²=0.25)

2

4

4.0 (e=0, e²=0)

3.5 (e=0.5, e²=0.25)

3

5

5.5 (e=-0.5, e²=0.25)

4.5 (e=0.5, e²=0.25)


Line A has SSE = 0.50 and Line B has SSE = 0.75, so by the least-squares criterion Line A fits these three points better, even though both lines pass reasonably close to the data.



When fitting a straight line to a set of (xi,yi)(x_i,y_i) data points, what quantity does the method of least squares minimize?

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

The figure shows five data points together with the least-squares line fitted to them, and at each point a red box whose area represents that point's squared residual (all boxes drawn to a common scale). Which point contributes the largest amount to the total SSE for this line?

The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.

The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.

Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point

Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

When fitting a straight line with an intercept to data containing at least two distinct observed xx-values, which statement best explains why the least-squares criterion squares each residual?

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors