Principal component analysis in this visualization
Put a line through the mean of a cloud of data and project every point onto it. How much the projections spread out depends on the line’s direction : their sample variance is , where is the sample covariance matrix: the two variances on the diagonal and the covariance off it (sums divided by ). Principal component analysis takes the direction in which this variance is largest as the first principal direction, and the line through the mean in that direction as the first principal component, PC1; PC2 is perpendicular to it. In the picture the red line is , the yellow ticks are the projections, and the thick yellow bar shows one standard deviation of them each way.
Along plus across stays the same
Each point’s squared distance from the mean splits, by Pythagoras, into a part along and a part across it. Adding up over the data, the variance along plus the variance across it equals the total variance, , whichever way points. So the direction with the most spread along it is also the one with the least left across it: among lines through the mean, PC1 makes the sum of squared perpendicular distances smallest. That is not the regression line of on , which makes the sum of squared vertical residuals smallest; for a loose cloud the two can be far apart.
Why the answer is an eigenvector
is symmetric, so it has two perpendicular eigenvectors with eigenvalues . Writing in terms of them, is a weighted average of and , largest at the first eigenvector and smallest at the second. So PC1 and PC2 are the eigenvectors of , the variances along them are and , and PC1 keeps the share of the total. In practice the components are often computed from the singular value decomposition of the centred data matrix, which gives the same directions.
Units change the answer
Variances depend on units. With heights in metres and weights in kilograms, the weights’ numbers vary far more, and PC1 lies almost along the weight axis for no other reason. When the variables are in different units, each is usually standardized first: subtract its mean, divide by its standard deviation. Then becomes the correlation matrix , and for two variables PC1 lies at 45° (or −45° when ) with ; when no direction stands out.
Things to notice
- A principal component is a line: its direction is fixed only up to sign.
- With many variables the same idea finds a few directions that hold most of the variance, so the data can be described with fewer numbers.
Controls
- Drag the red handle or turn the dial to turn the line; arrow keys on the handle turn it by 1°.