Estimating Model Parameters
Estimating regression coefficients from the design matrix
In a linear regression, the design matrix X holds an intercept column and predictor columns, while the response vector y holds the observed outcomes. The coefficient vector β describes a candidate fit. Ordinary least squares chooses its estimate β^ to make the sum of squared differences between observed and fitted responses as small as possible.
From residuals to coefficients
The residual vector is y−Xβ. Differentiating its squared length with respect to the coefficients and setting the derivative to zero gives the normal equations, XTXβ^=XTy. Thus each design column is orthogonal to the fitted residual vector. These equations do not require every residual to vanish.
If X has full column rank, XTX is invertible and β^=(XTX)−1XTy. If two design columns are identical, their separate coefficients cannot be identified from the fitted values. More observations alone do not resolve that exact redundancy.
Worked example
Take three observations with design rows (1,0), (1,1), and (1,2), and responses 1, 2, and 4. Then XTX=(3335) and XTy=(7,10)T. Solving the two equations gives an intercept of 5/6 and a slope of 3/2.
Check both equations: 3(5/6)+3(3/2)=7 and 3(5/6)+5(3/2)=10. The fitted values are 5/6, 7/3, and 23/6, so the residuals are 1/6, −1/3, and 1/6. Their sum and their predictor-weighted sum are zero, as the normal equations require.
Uncertainty and inference
Conditional on X, independent mean-zero errors with common variance σ2 give Cov(β^∣X)=σ2(XTX)−1. When n>p, the usual residual-variance estimate is s2=SSE/(n−p), where p counts fitted coefficients. A coefficient's standard error is the square root of s2 times the corresponding diagonal entry of the inverse matrix.
For a null value βj,0, the t statistic is (β^j−βj,0)/SE(β^j). With normal errors in addition to the stated assumptions, its exact conditional reference distribution has n−p degrees of freedom. This calculation describes sampling uncertainty; it does not turn a regression coefficient into a causal effect.
For a centered-predictor design with XᵀX = [[4,0],[0,20]] and Xᵀy = [8,30]ᵀ, what are the intercept and slope estimates?
A two-column design matrix has an intercept column of all ones and a second column also of all ones. Which statement about β̂ = (XᵀX)⁻¹Xᵀy is correct?