Biased and AntiBiased Variance Estimates 

Suppose S is a set of numbers whose mean value is X, and suppose x is an element of S. We wish to define the "variance" of x with respect to S as a measure of the degree to which x differs from the mean X. It turns out to be most useful to define the variance as the square of the difference between x and X. We'll denote this by V(xS) = (xX)^{2}. Furthermore, we define the variance of any subset s_{1} of S as the average of the variances of the elements of s_{1}. Thus, given a set s of n numbers x_{1}, x_{2}, ..., x_{n}, from a set S whose mean is X, the variance of s with respect to S is given by 

_{} 

It's important to note that the value of X in this equation is the mean of all of set S, not just the mean of the values of s. If, for some reason, we don't know the true mean of S we might try to apply formula (1) using an estimated mean based just on the values in s. Thus, if we define X' = (x_{1}+x_{2}+..+x_{n})/n, we could use this value in place of X in equation (1) to estimate the variance of s. However, this would result in a biased estimate, because X' is biased toward the elements of s. Thus each difference (x_{i}X') is slightly selfreferential, tending to underestimate the true variance of x_{i} with respect to the full set S. 

What if we try to eliminate the bias by simply removing x_{i} from X'? In other words, let's define X'_{i} as the average of the n1 measurements excluding x_{i}. At first we might think this would lead to an unbiased estimate of the variance, but that's not right, because by specifically excluding the measurement x_{i} from the mean when evaluating each term (x_{i}  X'_{i})^{2} we are effectively creating an antibiased formula, tending to overestimate the variance. What we need is something in between the biased and antibiased estimates. 

If we define the ordinary (biased) variance of s with respect to S as 

_{} 

and the "antibiased" mean variance as 

_{} 

then it's easy to see that 

_{} 

and so we have 

_{} 

Thus, the estimates V and V* are (sort of) duals of each other, and their geometric mean gives equation (5), which we recognize as the unbiased variance estimate of the underlying set S based on s. In fact, if we could think of some good simple reason why the unbiased estimate must equal _{}, this would constitute a simple derivation of the unbiased estimate. 

Of course, the idea of the "unbiased estimate" is that if we draw out a sample of n items from an unknown population and compute the variance for that sample using equation (2), then we take another sample of n and compute the variance for that sample, and so on, and then after awhile we take the mean of all these variances, the average would approach not V(SS) but rather [(n1)/n] V(SS). 

As an example, recall that the distribution of variances of nsamples from a normal distribution has a “chisquare" distribution with n1 degrees of freedom. Thus, in order to have a measure of variance that converges precisely on "s^{2}" for a normal distribution, we have to divide S(x_{i}  X)^{2} by n1 instead of n. In other words, we have to use the unbiased estimate given by equation (5). 

Incidentally, another way of expressing the unbiased variance estimate is to use a "weighted" mean X"_{i} defined as 

_{} 

where k is the "weight" assigned to xi to get an effectively unbiased estimate of the mean X. If we substitute X"_{i} in place of X'_{i} in equation (3) the result will equal the unbiased estimate if and only if 

_{} 

which implies that the correct "weight" for any given n is 

_{} 

Also, the lefthand factor shows that the estimate is unbiased for any weight k if the values of the n numbers are such that 

_{} 

Given any set of x values (which may be complex) we can create a "nullbiased" set just by adding one more number. We can also add more numbers while maintaining the above null bias condition, but after the 2nd number all the remaining numbers are identical. For example, given the set {0,1} we can add 

_{} 

to create a nullbias set with three elements, and then we can add 

_{} 

to create a nullbias set with four elements. Thereafter we can only increase the number of elements by adding duplicates of the 4th element. 
