Covariance Matrix — Definition, Formula & Example
A covariance matrix is a square table that shows how much each variable in a data set varies on its own and how each pair of variables varies together. The variances sit on the main diagonal, and the covariances fill the off-diagonal entries.
For a random vector with mean vector , the covariance matrix is the matrix , whose entry is . It is always symmetric () and positive semidefinite. Its sample estimate from observations is denoted .
Key Formula
Where:
- = Entry in row , column of the sample covariance matrix
- = Number of observations
- = Value of variable in observation
- = Sample mean of variable
Worked Example
Problem: Three observations of two variables are recorded: and . Find the sample covariance matrix.
Step 1: Find each mean. The deviations from the means are then for and for .
Step 2: Compute the variances for the diagonal. Divide by .
Step 3: Compute the covariance by multiplying paired deviations. This value fills both off-diagonal entries.
Answer: . The positive covariance of 7 tells you that and tend to increase together.
Why It Matters
Principal component analysis (PCA) finds the eigenvectors of the covariance matrix to reduce many variables to a few key directions. In finance, the variance of a portfolio with weight vector is , which is the core of Markowitz portfolio theory. The matrix also defines the shape of the multivariate normal distribution used throughout machine learning.
Common Mistakes
Mistake: Dividing by when computing a covariance matrix from sample data.
Correction: Use for a sample covariance matrix so that it is an unbiased estimate of . Divide by only when your data is the entire population. Check which convention your software uses. NumPy's np.cov uses by default.
