Linear Algebra · Principal component analysis

Principal component analysis

Principal component analysis finds the directions in which a cloud of data spreads most. For two variables, the first principal direction is the unit direction along which the projected data have the largest variance: an eigenvector of the covariance matrix for its largest eigenvalue, which is that variance. The first principal component line runs through the mean in that direction.

Principal component analysis in this visualization

Put a line through the mean of a cloud of data and project every point onto it. How much the projections spread out depends on the line’s direction uu: their sample variance is uTCuu^{\mathsf T} C u, where C=(sx2sxysxysy2)C = \begin{pmatrix} s_x^2 & s_{xy} \\ s_{xy} & s_y^2 \end{pmatrix} is the sample covariance matrix: the two variances on the diagonal and the covariance off it (sums divided by n−1n - 1). Principal component analysis takes the direction in which this variance is largest as the first principal direction, and the line through the mean in that direction as the first principal component, PC1; PC2 is perpendicular to it. In the picture the red line is uu, the yellow ticks are the projections, and the thick yellow bar shows one standard deviation of them each way.

Along plus across stays the same

Each point’s squared distance from the mean splits, by Pythagoras, into a part along uu and a part across it. Adding up over the data, the variance along uu plus the variance across it equals the total variance, sx2+sy2=tr⁡Cs_x^2 + s_y^2 = \operatorname{tr} C, whichever way uu points. So the direction with the most spread along it is also the one with the least left across it: among lines through the mean, PC1 makes the sum of squared perpendicular distances smallest. That is not the regression line of yy on xx, which makes the sum of squared vertical residuals smallest; for a loose cloud the two can be far apart.

Why the answer is an eigenvector

CC is symmetric, so it has two perpendicular eigenvectors with eigenvalues λ1≥λ2\lambda_1 \ge \lambda_2. Writing uu in terms of them, uTCuu^{\mathsf T} C u is a weighted average of λ1\lambda_1 and λ2\lambda_2, largest at the first eigenvector and smallest at the second. So PC1 and PC2 are the eigenvectors of CC, the variances along them are λ1\lambda_1 and λ2\lambda_2, and PC1 keeps the share λ1/(λ1+λ2)\lambda_1/(\lambda_1 + \lambda_2) of the total. In practice the components are often computed from the singular value decomposition of the centred data matrix, which gives the same directions.

Units change the answer

Variances depend on units. With heights in metres and weights in kilograms, the weights’ numbers vary far more, and PC1 lies almost along the weight axis for no other reason. When the variables are in different units, each is usually standardized first: subtract its mean, divide by its standard deviation. Then CC becomes the correlation matrix (1rr1)\begin{pmatrix} 1 & r \\ r & 1 \end{pmatrix}, and for two variables PC1 lies at 45° (or −45° when r<0r < 0) with λ=1±∣r∣\lambda = 1 \pm |r|; when r=0r = 0 no direction stands out.

Things to notice

  • A principal component is a line: its direction is fixed only up to sign.
  • With many variables the same idea finds a few directions that hold most of the variance, so the data can be described with fewer numbers.

Controls

  • Drag the red handle or turn the dial to turn the line; arrow keys on the handle turn it by 1°.

Related