Mathwords logoMathwords

Matrix Calculus — Definition, Formula & Example

Matrix calculus is a compact notation for taking derivatives when the input or output of a function is a vector or matrix. It packs many partial derivatives into one array, so a whole gradient can come from one short rule.

For a scalar function f:Rn→Rf:\mathbb{R}^n\to\mathbb{R}, the gradient ∇xf\nabla_{\mathbf{x}} f is the n×1n\times 1 vector with entries ∂f/∂xi\partial f/\partial x_i. For f:Rn→Rm\mathbf{f}:\mathbb{R}^n\to\mathbb{R}^m, the Jacobian is the m×nm\times n matrix with entries Jij=∂fi/∂xjJ_{ij}=\partial f_i/\partial x_j. For a scalar function of an m×nm\times n matrix XX, ∂f/∂X\partial f/\partial X is the m×nm\times n matrix with entries ∂f/∂Xij\partial f/\partial X_{ij}. Whether a gradient is written as a column or a row depends on the layout convention (denominator or numerator layout).

Key Formula

∇x(xTA x)=(A+AT)x\nabla_{\mathbf{x}}\left(\mathbf{x}^{T}A\,\mathbf{x}\right) = \left(A + A^{T}\right)\mathbf{x}
Where:
  • x\mathbf{x} = An n×1n\times 1 column vector of variables
  • AA = A constant n×nn\times n matrix, not necessarily symmetric
  • ATA^{T} = The transpose of AA
  • ∇x\nabla_{\mathbf{x}} = The gradient with respect to x\mathbf{x}, written as a column vector (denominator layout)

Worked Example

Problem: Let A=[2134]A=\begin{bmatrix}2&1\\3&4\end{bmatrix} and f(x)=xTA xf(\mathbf{x})=\mathbf{x}^{T}A\,\mathbf{x}. Use the matrix calculus rule to find ∇f\nabla f at x=[12]\mathbf{x}=\begin{bmatrix}1\\2\end{bmatrix}, then confirm it with ordinary partial derivatives.
Step 1: Because AA is not symmetric, you need A+ATA + A^{T}, not 2A2A.
A+AT=[2134]+[2314]=[4448]A + A^{T} = \begin{bmatrix}2&1\\3&4\end{bmatrix} + \begin{bmatrix}2&3\\1&4\end{bmatrix} = \begin{bmatrix}4&4\\4&8\end{bmatrix}
Step 2: Multiply by x\mathbf{x} to get the gradient in one matrix product.
∇f=[4448][12]=[4+84+16]=[1220]\nabla f = \begin{bmatrix}4&4\\4&8\end{bmatrix}\begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}4+8\\4+16\end{bmatrix} = \begin{bmatrix}12\\20\end{bmatrix}
Step 3: Check: expanding gives f=2x12+4x1x2+4x22f = 2x_1^2 + 4x_1x_2 + 4x_2^2. Take partial derivatives and evaluate at (1,2)(1, 2).
∂f∂x1=4x1+4x2=12,∂f∂x2=4x1+8x2=20\frac{\partial f}{\partial x_1} = 4x_1 + 4x_2 = 12, \qquad \frac{\partial f}{\partial x_2} = 4x_1 + 8x_2 = 20
Answer: ∇f(1,2)=[1220]\nabla f(1,2) = \begin{bmatrix}12\\20\end{bmatrix}. The one-line matrix rule matches the term-by-term partial derivatives.

Why It Matters

Training a neural network relies on gradients of a loss with respect to whole weight matrices, and backpropagation is the matrix chain rule applied layer by layer. In least-squares regression, the gradient of ∥Ax−b∥2\|A\mathbf{x}-\mathbf{b}\|^2 is 2AT(Ax−b)2A^{T}(A\mathbf{x}-\mathbf{b}). Setting it to zero gives the normal equations ATAx=ATbA^{T}A\mathbf{x}=A^{T}\mathbf{b}.

Common Mistakes

Mistake: Using the rule ∇(xTAx)=2Ax\nabla(\mathbf{x}^{T}A\mathbf{x}) = 2A\mathbf{x} for every matrix AA.
Correction: The shortcut 2Ax2A\mathbf{x} holds only when AA is symmetric. In the example, 2Ax2A\mathbf{x} gives (8,22)(8, 22) instead of the correct (12,20)(12, 20). Use (A+AT)x(A + A^{T})\mathbf{x} in general. Also check which layout convention your textbook uses, because the other convention gives the transposed result.

Related Terms

  • Multivariable Calculus — The partial derivatives this notation bundles into arrays
  • Calculus — The core rules of derivatives this notation extends
  • Adjugate — Its transpose gives the gradient of the determinant
  • Compatible Matrices — Dimension checks needed for every matrix chain rule product
  • Column of a Matrix — Each Jacobian column holds partial derivatives for one input variable
  • Jacobian — Matrix of first-order partial derivatives of vector functions