Method of Least Squares
The Method of Least Squares: Minimizing Squared Residuals
The method of least squares is the rule used to pick the "best" straight line through a scatter of points.
For any trial line y^=b0+b1x, each data point produces a residual ei=yi−y^i, the vertical gap between the observed value and the line. Different lines give different sets of residuals, so we need one number that scores how good a line is overall.
Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point
The least-squares criterion defines that score as the sum of squared residuals, often written SSE:
The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.
The "best" line, by definition, is the one that makes S(b0,b1) as small as possible. This discussion assumes that the model fits an intercept and that at least two observed x-values are distinct.
Squaring, rather than adding the residuals directly, prevents positive and negative residuals from cancelling, so a line cannot look artificially perfect just because its errors balance to zero.
Under those conditions, SSE is a strictly convex quadratic in b0 and b1, so it has a unique minimizing pair; smoothness enables differentiation to locate it. Adding absolute residuals also prevents cancellation, but it defines a different fitting method, least absolute deviations.
Worked example
Consider the points (1,2),(2,4),(3,5). Compare two candidate lines using SSE:
x | y | Line A: y^=1+1.5x | Line B: y^=1.5+1x |
|---|---|---|---|
1 | 2 | 2.5 (e=-0.5, e²=0.25) | 2.5 (e=-0.5, e²=0.25) |
2 | 4 | 4.0 (e=0, e²=0) | 3.5 (e=0.5, e²=0.25) |
3 | 5 | 5.5 (e=-0.5, e²=0.25) | 4.5 (e=0.5, e²=0.25) |
Line A has SSE = 0.50 and Line B has SSE = 0.75, so by the least-squares criterion Line A fits these three points better, even though both lines pass reasonably close to the data.
The figure shows five data points together with the least-squares line fitted to them, and at each point a red box whose area represents that point's squared residual (all boxes drawn to a common scale). Which point contributes the largest amount to the total SSE for this line?
The least-squares line for these five points, with each point's squared residual drawn as a box; the box area equals (residual)², all boxes to a common scale.
Sum of squared residuals for trial slopes, with each line constrained to pass through the mean point
When fitting a straight line with an intercept to data containing at least two distinct observed x-values, which statement best explains why the least-squares criterion squares each residual?