Estimating regression coefficients from the design matrix

In a linear regression, the design matrix XX holds an intercept column and predictor columns, while the response vector yy holds the observed outcomes. The coefficient vector β\beta describes a candidate fit. Ordinary least squares chooses its estimate β^\hat\beta to make the sum of squared differences between observed and fitted responses as small as possible.

From residuals to coefficients

The residual vector is y−Xβy-X\beta. Differentiating its squared length with respect to the coefficients and setting the derivative to zero gives the normal equations, XTXβ^=XTyX^{\mathsf T}X\hat\beta=X^{\mathsf T}y. Thus each design column is orthogonal to the fitted residual vector. These equations do not require every residual to vanish.


If XX has full column rank, XTXX^{\mathsf T}X is invertible and β^=(XTX)−1XTy\hat\beta=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y. If two design columns are identical, their separate coefficients cannot be identified from the fitted values. More observations alone do not resolve that exact redundancy.

Worked example

Take three observations with design rows (1,0)(1,0), (1,1)(1,1), and (1,2)(1,2), and responses 11, 22, and 44. Then XTX=(3335)X^{\mathsf T}X=\begin{pmatrix}3&3\\3&5\end{pmatrix} and XTy=(7,10)TX^{\mathsf T}y=(7,10)^{\mathsf T}. Solving the two equations gives an intercept of 5/65/6 and a slope of 3/23/2.


Check both equations: 3(5/6)+3(3/2)=73(5/6)+3(3/2)=7 and 3(5/6)+5(3/2)=103(5/6)+5(3/2)=10. The fitted values are 5/65/6, 7/37/3, and 23/623/6, so the residuals are 1/61/6, −1/3-1/3, and 1/61/6. Their sum and their predictor-weighted sum are zero, as the normal equations require.

Uncertainty and inference

Conditional on XX, independent mean-zero errors with common variance σ2\sigma^2 give Cov⁡(β^∣X)=σ2(XTX)−1\operatorname{Cov}(\hat\beta\mid X)=\sigma^2(X^{\mathsf T}X)^{-1}. When n>pn>p, the usual residual-variance estimate is s2=SSE/(n−p)s^2=\mathrm{SSE}/(n-p), where pp counts fitted coefficients. A coefficient's standard error is the square root of s2s^2 times the corresponding diagonal entry of the inverse matrix.


For a null value βj,0\beta_{j,0}, the t statistic is (β^j−βj,0)/SE⁡(β^j)(\hat\beta_j-\beta_{j,0})/\operatorname{SE}(\hat\beta_j). With normal errors in addition to the stated assumptions, its exact conditional reference distribution has n−pn-p degrees of freedom. This calculation describes sampling uncertainty; it does not turn a regression coefficient into a causal effect.



For a design matrix X with rows (1,2), (1,4), and (1,6), and response vector y, which equation defines the ordinary least-squares coefficient vector β̂?

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

For a centered-predictor design with XᵀX = [[4,0],[0,20]] and Xᵀy = [8,30]ᵀ, what are the intercept and slope estimates?

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors

A two-column design matrix has an intercept column of all ones and a second column also of all ones. Which statement about β̂ = (XᵀX)⁻¹Xᵀy is correct?

Choose an answer

Create a free account to get started

Unlock full course notes by signing up

  • Video lessons taught by top educators
  • Practice questions with guided solutions
  • Written review notes for reference
  • Expert answers anytime from course tutors