Mathwords logoMathwords

Covariance Matrix — Definition, Formula & Example

A covariance matrix is a square table that shows how much each variable in a data set varies on its own and how each pair of variables varies together. The variances sit on the main diagonal, and the covariances fill the off-diagonal entries.

For a random vector X=(X1,…,Xp)T\mathbf{X} = (X_1, \dots, X_p)^T with mean vector μ\boldsymbol{\mu}, the covariance matrix is the p×pp \times p matrix Σ=E[(X−μ)(X−μ)T]\Sigma = E\big[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T\big], whose (j,k)(j,k) entry is Cov⁡(Xj,Xk)\operatorname{Cov}(X_j, X_k). It is always symmetric (Σ=ΣT\Sigma = \Sigma^T) and positive semidefinite. Its sample estimate from nn observations is denoted SS.

Key Formula

sjk=1n−1∑i=1n(xij−xˉj)(xik−xˉk)s_{jk} = \frac{1}{n-1} \sum_{i=1}^{n} (x_{ij} - \bar{x}_j)(x_{ik} - \bar{x}_k)
Where:
  • sjks_{jk} = Entry in row jj, column kk of the sample covariance matrix SS
  • nn = Number of observations
  • xijx_{ij} = Value of variable jj in observation ii
  • xˉj\bar{x}_j = Sample mean of variable jj

Worked Example

Problem: Three observations of two variables are recorded: X=2,4,6X = 2, 4, 6 and Y=1,3,8Y = 1, 3, 8. Find the sample covariance matrix.
Step 1: Find each mean. The deviations from the means are then −2,0,2-2, 0, 2 for XX and −3,−1,4-3, -1, 4 for YY.
xˉ=2+4+63=4,yˉ=1+3+83=4\bar{x} = \frac{2+4+6}{3} = 4, \qquad \bar{y} = \frac{1+3+8}{3} = 4
Step 2: Compute the variances for the diagonal. Divide by n−1=2n - 1 = 2.
sX2=(−2)2+02+222=4,sY2=(−3)2+(−1)2+422=13s_X^2 = \frac{(-2)^2 + 0^2 + 2^2}{2} = 4, \qquad s_Y^2 = \frac{(-3)^2 + (-1)^2 + 4^2}{2} = 13
Step 3: Compute the covariance by multiplying paired deviations. This value fills both off-diagonal entries.
sXY=(−2)(−3)+(0)(−1)+(2)(4)2=142=7s_{XY} = \frac{(-2)(-3) + (0)(-1) + (2)(4)}{2} = \frac{14}{2} = 7
Answer: S=(47713)S = \begin{pmatrix} 4 & 7 \\ 7 & 13 \end{pmatrix}. The positive covariance of 7 tells you that XX and YY tend to increase together.

Why It Matters

Principal component analysis (PCA) finds the eigenvectors of the covariance matrix to reduce many variables to a few key directions. In finance, the variance of a portfolio with weight vector w\mathbf{w} is wTΣw\mathbf{w}^T \Sigma \mathbf{w}, which is the core of Markowitz portfolio theory. The matrix also defines the shape of the multivariate normal distribution used throughout machine learning.

Common Mistakes

Mistake: Dividing by nn when computing a covariance matrix from sample data.
Correction: Use n−1n - 1 for a sample covariance matrix so that it is an unbiased estimate of Σ\Sigma. Divide by nn only when your data is the entire population. Check which convention your software uses. NumPy's np.cov uses n−1n - 1 by default.

Related Terms

  • Matrix — The array that holds the variances and covariances
  • Covariance — Each off-diagonal entry is a covariance
  • Bivariate Data — Two-variable data, giving a 2×2 covariance matrix
  • Scatterplot — Visualizes the covariance between two variables
  • Univariate Data — One variable; its covariance matrix is just the variance
  • Variance — Diagonal entries of the matrix are variances
  • Eigenvector — Eigenvectors of it give principal directions of variation