p-value

A hypothesis test asks whether data would be surprising under a default claim, the null hypothesis $H_0$. Here $H_0$ says that a machine fills bags with $\mu = 500$ g on average; bag weights are independent and normal with $\sigma = 10$ g, and a study weighs $n$ bags and takes their mean $\bar x$. The p-value is the probability, computed as if $H_0$ were true, of a mean at least as extreme as the observed one, in the direction named by the alternative hypothesis $H_1$. A test at level $\alpha = 0.05$ raises an alarm when $p < 0.05$.

What exactly is p the probability of? Under $H_0$ the mean $\bar X$ is normal with mean 500 and standard error $10/\sqrt{n}$, so the observed mean sits at $z = (\bar x - 500)/(10/\sqrt{n})$. For $H_1\!: \mu < 500$ the p-value is the area to the left of $z$, $\Phi(z)$; for $\mu > 500$ it is the area to the right; for $\mu \neq 500$ it is both tails, $2\Phi(-|z|)$. The reruns show the same thing by counting: on a machine that truly fills 500 g, the share of studies whose mean is at least this extreme settles near p. The direction has to be chosen before the data are seen. Picking the smaller of the two one-sided p-values afterwards raises an alarm 10% of the time when $H_0$ is true, twice the stated rate. With the direction fixed in advance, p is uniformly distributed between 0 and 1 under $H_0$, so the test raises a false alarm exactly a fraction $\alpha$ of the time.

Why is p not the probability that $H_0$ is true? It is computed entirely inside the world of $H_0$: it is $P(\text{a mean at least this extreme} \mid H_0)$, not $P(H_0 \mid \text{the data})$. Turning it around needs two quantities that p never uses: how common short machines are to begin with, and the power, the chance that the test raises an alarm on a short machine. With 25 bags and a 3 g shortfall the power is about $\Phi(1.5 - 1.645)$. In the odds form of Bayes' theorem, an alarm multiplies the odds of short against good by power ÷ $\alpha$, about 8.85 here, taking prior odds of 1 : 4 to about 2.2 : 1. After the same alarm, the share of good machines among the alarms is about 2.7% when 80% of machines are short and about 68% when 5% are. These shares use only the fact that an alarm was raised; the value of $\bar x$ itself says more.

Why can a tiny difference be significant? For a fixed observed shortfall, $|z| = |\bar x - 500|\sqrt{n}/\sigma$ grows with $\sqrt{n}$, so the lower-tail p gets smaller as more bags are weighed: four times as many bags doubles $|z|$. In the lower-tail test, half a gram below 500 is significant at $\alpha = 0.05$ from 1083 bags on. A small p says that the data would be unusual if $\mu$ were exactly 500; it does not say that the difference matters. The converse fails too: a real shortfall often raises no alarm, so $p > 0.05$ does not show that the machine is fine. Report the size of the difference with a confidence interval: the two-sided test at $\alpha = 0.05$ rejects $\mu = 500$ exactly when the 95% interval $\bar x \pm z^*\sigma/\sqrt{n}$ ($z^* \approx 1.96$) leaves out 500.