Orthogonal projection

Orthogonal projection finds the point of a subspace that is closest to a given vector. Here the subspace $S$ is a plane or a line through the origin O, and the vector is $\vec b$. Its projection $\vec p$ lies in $S$, and the error $\vec e = \vec b - \vec p$ meets $S$ at a right angle. You can move $\vec b$ and reshape $S$, compare $\vec p$ with any other point of $S$, and see when the shadows of $\vec b$ on two axes add up to $\vec p$.

Why does the right angle give the closest point? Take any other point $q$ in $S$. The segment from $q$ to $\vec p$ lies in $S$, so it is perpendicular to $\vec e$. The points $\vec b$, $\vec p$ and $q$ form a right triangle with the right angle at $\vec p$, and Pythagoras gives

So $\vec b$ is farther from $q$ than from $\vec p$, unless $q$ is $\vec p$ itself. It works the other way too. If $\vec e$ leaned toward some direction in $S$, moving $\vec p$ slightly that way would bring it closer to $\vec b$. In a plane, then, the closest point is the one where $\vec e$ is perpendicular to both $\vec u$ and $\vec w$. On a line, being perpendicular to $\vec u$ is enough.

Finding $\vec p$ comes down to these right angles. On a line, $\vec p$ is a multiple of $\vec u$, and the condition $\vec e\cdot\vec u = 0$ fixes the multiple:

In a plane, write $\vec p = c_1\vec u + c_2\vec w$. The two conditions $\vec e\cdot\vec u = 0$ and $\vec e\cdot\vec w = 0$ become two linear equations for $c_1$ and $c_2$, called the normal equations:

The readout gives $\vec p$ both as coordinates and as a combination of $\vec u$ and $\vec w$. Drag $\vec u$ and $\vec w$ around within the same plane and the coefficients change, but $\vec p$ does not. The projection depends only on the subspace, not on the vectors used to describe it.

When $\vec u$ and $\vec w$ are perpendicular, the two equations come apart. Each coefficient then depends on one axis only, and $\vec p$ is simply the sum of the shadows of $\vec b$ on the two axes:

On slanted axes this shortcut usually fails. Each shadow is taken as if the other axis were not there, so their sum $s$ usually lands away from $\vec p$. When it does, the readout shows $\vec b - s$ missing at least one of the two right angles. Make w ⟂ u replaces $\vec w$ with its part perpendicular to $\vec u$, which is one step of the Gram–Schmidt process. The plane is still the same plane, but now its two axes are perpendicular, and the shadows add up to $\vec p$ for every $\vec b$.

The diagonal line through $(1, 1, 1)$ gives a familiar example. Projecting $\vec b$ onto it replaces every entry of $\vec b$ with their mean, and $|\vec e|^2$ is the sum of squared deviations from that mean. Least-squares regression uses the same kind of projection.