Mathwords logoMathwords

Singular Value Decomposition — Definition, Formula & Examples

Singular Value Decomposition (SVD) is a way to factor any matrix AA into three special matrices: an orthogonal matrix UU, a diagonal matrix Σ\Sigma of non-negative values called singular values, and the transpose of another orthogonal matrix VTV^T. It works for any m×nm \times n matrix, even non-square ones.

For any real m×nm \times n matrix AA, there exist orthogonal matrices U∈Rm×mU \in \mathbb{R}^{m \times m} and V∈Rn×nV \in \mathbb{R}^{n \times n}, and a diagonal matrix Σ∈Rm×n\Sigma \in \mathbb{R}^{m \times n} with non-negative entries σ1≥σ2≥⋯≥σmin⁡(m,n)≥0\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_{\min(m,n)} \geq 0 on its main diagonal, such that A=UΣVTA = U\Sigma V^T. The diagonal entries σi\sigma_i are the singular values of AA, equal to the square roots of the eigenvalues of ATAA^T A.

Key Formula

A=UΣVTA = U \Sigma V^T
Where:
  • AA = The original m × n matrix being decomposed
  • UU = An m × m orthogonal matrix whose columns are left singular vectors
  • Σ\Sigma = An m × n diagonal matrix of singular values σ₁ ≥ σ₂ ≥ … ≥ 0
  • VTV^T = The transpose of an n × n orthogonal matrix whose columns are right singular vectors

How It Works

To compute the SVD of AA, first form ATAA^T A and find its eigenvalues λi\lambda_i. The singular values are σi=λi\sigma_i = \sqrt{\lambda_i}, arranged in decreasing order along the diagonal of Σ\Sigma. The columns of VV are the corresponding eigenvectors of ATAA^T A (orthonormalized). The columns of UU are found from ui=1σiAviu_i = \frac{1}{\sigma_i} A v_i for each nonzero singular value. The rank of AA equals the number of nonzero singular values.

Worked Example

Problem: Find the singular values of the matrix A = [[3, 0], [0, 4]].
Step 1: Compute ATAA^T A.
ATA=[3004]T[3004]=[90016]A^T A = \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix}^T \begin{bmatrix} 3 & 0 \\ 0 & 4 \end{bmatrix} = \begin{bmatrix} 9 & 0 \\ 0 & 16 \end{bmatrix}
Step 2: Find the eigenvalues of ATAA^T A. Since it is diagonal, the eigenvalues are the diagonal entries: λ1=16\lambda_1 = 16 and λ2=9\lambda_2 = 9.
λ1=16,λ2=9\lambda_1 = 16,\quad \lambda_2 = 9
Step 3: The singular values are the square roots of the eigenvalues, in decreasing order.
σ1=16=4,σ2=9=3\sigma_1 = \sqrt{16} = 4,\quad \sigma_2 = \sqrt{9} = 3
Answer: The singular values of AA are σ1=4\sigma_1 = 4 and σ2=3\sigma_2 = 3, so Σ=[4003]\Sigma = \begin{bmatrix} 4 & 0 \\ 0 & 3 \end{bmatrix}.

Why It Matters

SVD is foundational in data science and machine learning — it powers principal component analysis (PCA), image compression, and recommendation systems like those used by streaming services. In numerical linear algebra courses, it provides the most numerically stable way to determine the rank of a matrix and solve least-squares problems.

Common Mistakes

Mistake: Confusing singular values with eigenvalues of AA itself.
Correction: Singular values are the square roots of the eigenvalues of ATAA^T A, not of AA. They are always non-negative, whereas eigenvalues of AA can be negative or complex.