Statistical power

The power of a test is the probability that it rejects the null hypothesis $H_0$ when a particular alternative is true. Here $H_0$ says that a machine fills bags with $\mu = 500$ g on average; bag weights are independent and normal with $\sigma = 10$ g, and a study weighs $n$ bags and rejects $H_0$ at level $\alpha$ when their mean $\bar x$ falls past a cut-off. The power at a true mean $\mu$ is the area of the true curve beyond that cut-off; the rest, $\beta$, is the chance of missing the difference, a type II error. At $\mu = 500$ there is nothing to miss, and the power is $\alpha$ itself.

What does power depend on? For the test of $H_1\!: \mu < 500$ with a shortfall $\delta = 500 - \mu$, the power is $\Phi(\delta\sqrt{n}/\sigma - z_{1-\alpha})$. A larger gap or more bags raise it; a stricter $\alpha$ moves the cut-off away from 500 and lowers it. Because $n$ enters only through $\sqrt{n}$, the number of bags needed for a given power grows like $1/\delta^2$. The direction matters as much: if the machine actually overfills, the power of the test of $\mu < 500$ falls toward 0, and more bags only push it lower. A two-sided test splits $\alpha$ between the tails, so it catches a gap on either side, but on the side a one-sided test watches, it has less power than that test.

Why does a non-significant result not show that there is no difference? Because a test with low power usually misses a real one: with 25 bags at $\alpha = 0.05$, the one-sided test catches a machine 2 g short only about a quarter of the time, so most studies of it end non-significant. What a non-significant result is worth depends on the power against a difference worth finding, and that is worked out before the data are seen: choose the smallest gap that matters, then an $n$ that gives enough power, often 80%.

Why do significant estimates exaggerate the difference on average when power is low? Take a machine that really underfills, tested for underfilling. Only studies whose mean happens to fall far below 500 pass the cut-off, so the significant ones are a selected group: their average estimate is the mean of the true curve beyond the cut-off, a larger shortfall than the true one. The cut-off itself sets a floor: with 25 bags at $\alpha = 0.05$, a significant result must estimate a shortfall of more than about 3.29 g, whatever the true shortfall is. With $n$ and $\alpha$ fixed, the smaller the true shortfall, the lower the power and the larger the factor by which significant estimates overstate it on average, a type M (magnitude) error; in a two-sided test some significant results even point the wrong way, a type S (sign) error. A body of research that reports only significant results carries this bias.