Probability density function

A probability density function $f$ describes a continuous random variable $X$ through areas: $P(a \le X \le b) = \int_a^b f(x)\,dx$, where $f \ge 0$ and the total area is 1. Here the line is cut into bins of width $w$, and each bar is a bin’s exact probability, either as it is or divided by $w$, drawn over $f$; below it is the cumulative distribution function $F$. With the bars as densities, halving $w$ brings them onto $f$; there are five distributions to try, one of them drawn by hand.

What does the height $f(x)$ measure? Probability per unit length, not probability. A bin’s probability divided by its width is the average of $f$ over the bin, so where $f$ is continuous at $x$, the bar around $x$ tends to $f(x)$ as $w$ shrinks, and a short interval has $P(x \le X \le x + \Delta x) \approx f(x)\,\Delta x$. That is why a density can exceed 1 even though no probability does: probability is the area, and a tall, narrow region can have a small area. A single point is an interval of width 0, so $P(X = x) = 0$ whatever $f(x)$ is, and for such a variable $\le$ and $<$ give the same probabilities.

How do the density and the cumulative distribution function fit together? $F(x) = P(X \le x)$ is the area under $f$ to the left of $x$, so the area over $[a, b]$ above is the rise $F(b) - F(a)$ below. A bin’s probability is the rise of $F$ across the bin, so the bar divided by $w$ is the slope of the secant of $F$ over that bin. As $w$ shrinks, the secant turns toward the tangent: where $f$ is continuous, $F'(x) = f(x)$, which is the fundamental theorem of calculus. Where $f$ jumps, as at the ends of the uniform, $F$ has a corner, and its slopes from the two sides are the values of $f$ on the two sides. Changing $f$ at a single point changes no area, so no probability changes either.

Why is a histogram divided by its bin width before it is compared with a density? Without the division the bars are probabilities, and they sink toward 0 as the bins narrow, not toward $f$. Divided by $w$, the bars over the whole line have total area 1 whatever the width, so histograms with different bins can be compared with each other and laid over $f$. A histogram of data works the same way, with each bin’s share of the sample in place of its exact probability.