Linear regression

Simple linear regression predicts $y$ from $x$ with the least-squares line $\hat y = a + bx$: the straight line with the smallest sum of squared residuals, where a point's residual $e = y - \hat y$ is how far it lies above the line (negative below). With a few points each residual carries a square, so the sum is an area you can see. You can shift and turn a line of your own to look for the least, and compare the fitted line with a simulated cloud of 1000 points.

Why does the line pass through the center $(\bar x, \bar y)$? Write any line as $y = h + b(x - \bar x)$, with $h$ its height above $\bar x$. Its residuals add up to $n(\bar y - h)$, and its sum of squares splits in two:

Here $S_{xx}$, $S_{yy}$ and $S_{xy}$ are the sums of squared and multiplied deviations from the means. The first part depends only on the height and the second only on the slope, so at any slope the best height is $\bar y$: the best line passes through the center, and its residuals add up to 0. Turned about the center, the sum follows the parabola $P(b)$. When the $x$ values are not all equal, its single lowest point is at $b = S_{xy}/S_{xx}$.

What does $R^2$ measure? The flat line $y = \bar y$ is the parabola at slope 0, and its sum of squares is the total variation $\mathrm{SST} = S_{yy}$. The fitted line leaves $\mathrm{SSE}$; when the $y$ values vary, $R^2 = 1 - \mathrm{SSE}/\mathrm{SST}$ is the share of the variation in $y$ that it accounts for. When both coordinates vary, $R^2 = r^2$, so a correlation of 0.5 accounts for only a quarter. Accounting for is fitting, not causing.

Why is the line flatter than the cloud? In standard units its predictions are $\hat z_y = r\,z_x$, so when $|r| < 1$ a point away from $\bar x$ gets a prediction fewer standard deviations from $\bar y$ than it lies from $\bar x$. In an oval cloud such as the simulated one, the SD line, with slope $\pm s_y/s_x$, runs along the cloud as the eye sees it; the regression line is flatter. Nothing pulls the data back: the effect holds for any such data, and predicting $x$ from $y$ shows it the other way, with a second line through the center that is steeper on the plot. In such a cloud, averages of $y$ in narrow strips of $x$ show the effect, close to the regression line but not exactly on it. The name comes from Francis Galton's 1886 study of heights: the children of very tall or very short parents were on average less extreme.

When can the line mislead? Raising one point by 1 raises its own fitted value by its leverage $h = 1/n + (x - \bar x)^2/S_{xx}$, which grows with the distance from $\bar x$. A point alone far out in $x$ can have leverage near 1: the line follows it, so even when it disagrees with every other point its own residual can look small, and even in a large cloud such a point can turn the line. And $r$ can be close to 1 while the residuals run positive at both ends and negative in the middle, a curve that no straight line follows.