Unbiased Estimators
Unbiased Estimators
An estimator is a rule built from sample data and used to guess an unknown population parameter such as μ, p or σ2. Different random samples give different numbers, so an estimator θ^ is itself a random variable with a sampling distribution. It is unbiased for θ when the mean of that distribution is the parameter itself:
Approximate sampling distributions of two competing estimators of the same parameter
The bias is Bias(θ^)=E(θ^)−θ, so an unbiased estimator has bias zero. The phrase "for every possible value of θ" carries weight: an estimator that happens to give the right answer on average at one particular θ but not at others is not unbiased.
Two target diagrams, one for each estimator. Each dot is one of eight equally likely possible values in that estimator’s complete sampling distribution at the shown θ; the offsets from θ stay the same for every possible θ. The crosshair marks the true parameter value. The left panel belongs to Estimator 1 and the right panel to Estimator 2.
Bias is not accuracy, and accuracy is not precision
Bias describes where the sampling distribution is centred. Variance describes how widely it spreads. The two are independent: an estimator can be centred on the target and still scatter widely, or be tightly clustered and consistently off to one side.
Neither property alone tells you how good a single estimate will be, which is why the mean squared error combines them as MSE=Variance+Bias2.
Linear combinations
Expectation is linear, so for any constants ci and observations with E(Xi)=μ,
which equals μ for every μ exactly when the coefficients sum to one. That single condition settles a whole family of questions. For instance μ^=52X1+53X2 has coefficients summing to 52+53=1, so it is unbiased for μ even though it weights the two observations unequally. Weighting unequally does change the variance, which is a separate question.
Three standard results
- The sample mean Xˉ is unbiased for μ, since its coefficients are 1/n each and sum to one.
- The sample proportion p^=X/n is unbiased for p.
- For independent identically distributed observations with finite variance, the sample variance S2=n−11∑(Xi−Xˉ)2 is unbiased for σ2, whereas the version dividing by n is biased: E[n1∑(Xi−Xˉ)2]=nn−1σ2. The reason is that the deviations are taken about Xˉ rather than about μ, and Xˉ is by construction the value that makes that sum of squares as small as possible, so the sum is systematically too small.
Worked example
Suppose the true variance is σ2=9 and a student divides by n=5 rather than n−1. The bias is
so on average this estimator falls short of σ2 by 1.8 units. Notice that the bias shrinks as n grows: the estimator is biased at every sample size but becomes asymptotically unbiased.
Let X1,X2,X3 be independent random variables drawn from a population with mean μ. Which of the following estimators of μ is unbiased?
Let X1,X2,X3 be independent observations with E(Xi)=μ for each i. Consider the estimator μ^=aX1+2aX2+3aX3, where a is a constant. Find the value of a that makes μ^ an unbiased estimator of μ. Give your answer as a decimal to four places.