Sampling distribution

A statistic is a number computed from a sample, such as its mean or median. Draw independent samples of the same size $n$ from one population again and again, compute the statistic each time, and the values form its sampling distribution. Its expected value minus the true value is the bias; its standard deviation is the standard error. The three layers keep three distributions apart: the population, the $n$ values of one sample, and the sampling distribution, one dot per sample.

Why does one sample stay wide while the means close in? A sample is a rough copy of the population, so as $n$ grows its standard deviation $s$ settles near the population’s $\sigma$ rather than shrinking. The mean of $n$ independent values behaves differently: values above and below the centre partly cancel, and for any population with a finite variance $\operatorname{Var}(\bar x) = \sigma^2/n$, so the standard error of the mean is $\sigma/\sqrt{n}$. Four times the sample size halves it. The standard error is therefore not the spread of a sample: it says how much the mean would move from one sample to the next. Why the means also take a bell shape is the central limit theorem.

Why does the sample variance divide by $n - 1$? Every sample satisfies $\sum (x - \mu)^2 = \sum (x - \bar x)^2 + n(\bar x - \mu)^2$. The sum of squared deviations is smallest when measured from the sample’s own mean; it falls short of the sum measured from the true mean by $n(\bar x - \mu)^2$, whose average is $n$ times the variance of $\bar x$: exactly $\sigma^2$. So $\sum (x - \bar x)^2$ averages $(n - 1)\sigma^2$, dividing by $n - 1$ makes $s^2$ unbiased, and dividing by $n$ falls short by $\sigma^2/n$ on average. The correction is for the variance only: because the square root bends down, $s$ itself still underestimates $\sigma$ on average (for normal data with $n = 5$, by about 6%). The maximum misses for a plainer reason: with the uniform population every value is below 100, so the largest value of a sample falls short of 100 on average.

Is the mean always the better estimate of the centre? It depends on the population. For a symmetric population the mean and the median are both centred on the true centre, so the better one is the one with the smaller standard error. With normal data the median’s is larger, for large $n$ about $\sqrt{\pi/2} \approx 1.25$ times the mean’s; with Laplace data, whose tails are heavier, it is smaller, about $1/\sqrt{2} \approx 0.71$ times. One extreme value moves the mean but barely moves the median, and heavy tails produce extreme values often. For a skewed population the two may estimate different numbers: the exponential population here has mean 50 and median $40 + 10\ln 2 \approx 46.93$.