Linear Algebra · Singular value decomposition

Singular value decomposition

A real 2 × 2 matrix can be written as A = UΣVᵀ: a rotation (or reflection) Vᵀ, a stretch Σ along the axes by the singular values σ₁ ≥ σ₂ ≥ 0, and another rotation (or reflection) U. Geometrically, A maps the unit circle to an ellipse whose axes come from two perpendicular directions.

Singular value decomposition in this visualization

A linear transformation of the plane maps the unit circle to an ellipse (flattened to a segment or a point when AA is singular). As vv goes round the circle, the length ∣Av∣|Av| is largest, σ1\sigma_1, in one direction v1v_1 and smallest, σ2\sigma_2, in the direction v2v_2 perpendicular to it. These are the singular values, and Av1=σ1u1Av_1 = \sigma_1 u_1, Av2=σ2u2Av_2 = \sigma_2 u_2 with unit vectors u1⊥u2u_1 \perp u_2 along the ellipse’s axes. Putting the columns together, A=UΣVTA = U\Sigma V^{\mathsf T}. In the picture the red arrows show a direction vv and the one perpendicular to it. The yellow arrows show their images under AA.

Why the right angle survives only there

The squared length is ∣Av∣2=vT(ATA)v|Av|^2 = v^{\mathsf T}(A^{\mathsf T}A)v, and ATAA^{\mathsf T}A is symmetric, so it has perpendicular eigenvectors: they are v1,v2v_1, v_2, with eigenvalues σ12,σ22\sigma_1^2, \sigma_2^2. Their images are perpendicular too, because Av1⋅Av2=v1TATAv2=σ22 v1⋅v2=0Av_1 \cdot Av_2 = v_1^{\mathsf T}A^{\mathsf T}Av_2 = \sigma_2^2\, v_1 \cdot v_2 = 0. When σ1>σ2>0\sigma_1 > \sigma_2 > 0, this pair (up to signs) is the only perpendicular pair whose images stay perpendicular; the others come out skewed. For A=(3045)A = \begin{pmatrix} 3 & 0 \\ 4 & 5 \end{pmatrix}, (1,0)(1, 0) and (0,1)(0, 1) go to (3,4)(3, 4) and (0,5)(0, 5), only 36.87° apart, and as vv turns the angle runs between 36.87° and 143.13°. When σ1=σ2\sigma_1 = \sigma_2 (a rotation or reflection times a positive number), every pair keeps its right angle; when σ2=0\sigma_2 = 0, one image is the zero vector and there is no angle to speak of.

Turn, stretch, turn

Read from the right, VTV^{\mathsf T} turns v1,v2v_1, v_2 onto the coordinate axes, Σ\Sigma stretches along the axes by σ1,σ2\sigma_1, \sigma_2, and UU turns the result onto u1,u2u_1, u_2 (here UU also flips one axis when det⁡A<0\det A < 0). The frames between the steps are one path chosen for display; only the steps themselves are the factors. Areas scale by σ1σ2=∣det⁡A∣\sigma_1\sigma_2 = |\det A|. Unlike diagonalization, this works for every matrix: the shear (1101)\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} has no eigenbasis but has singular values φ≈1.618\varphi \approx 1.618 and 1/φ≈0.6181/\varphi \approx 0.618. For a symmetric matrix with positive eigenvalues the two factorizations coincide.

What the singular values tell

With one zero singular value the plane collapses onto a line; with two it collapses to a point: the number of nonzero singular values is the rank. When σ2>0\sigma_2 > 0, the ratio σ1/σ2\sigma_1/\sigma_2 says how elongated the ellipse is, and so how much solving with AA can magnify relative errors. Applied to a table of data, the same directions are the principal components of principal component analysis.

Things to notice

  • Each pair vi,uiv_i, u_i can be flipped together; when σ1=σ2\sigma_1 = \sigma_2 the directions are not unique at all.
  • Every real m × n matrix has a singular value decomposition, with as many singular values as the smaller of m and n; this page shows the 2 × 2 case.

Controls

  • Drag the tip of v round the circle or turn the dial; turn the Steps wheel to watch the three factors one at a time.

Related