Matrix calculus is a compact notation for taking derivatives when the input or output of a function is a vector or matrix. It packs many partial derivatives into one array, so a whole gradient can come from one short rule.
For a scalar function f:Rn→R, the gradient ∇xf is the n×1 vector with entries ∂f/∂xi. For f:Rn→Rm, the Jacobian is the m×n matrix with entries Jij=∂fi/∂xj. For a scalar function of an m×n matrix X, ∂f/∂X is the m×n matrix with entries ∂f/∂Xij. Whether a gradient is written as a column or a row depends on the layout convention (denominator or numerator layout).
Key Formula
∇x(xTAx)=(A+AT)x Where:- x = An n×1 column vector of variables
- A = A constant n×n matrix, not necessarily symmetric
- AT = The transpose of A
- ∇x = The gradient with respect to x, written as a column vector (denominator layout)
Worked Example
Problem: Let A=[2314] and f(x)=xTAx. Use the matrix calculus rule to find ∇f at x=[12], then confirm it with ordinary partial derivatives. Step 1: Because A is not symmetric, you need A+AT, not 2A.A+AT=[2314]+[2134]=[4448] Step 2: Multiply by x to get the gradient in one matrix product.∇f=[4448][12]=[4+84+16]=[1220] Step 3: Check: expanding gives f=2x12+4x1x2+4x22. Take partial derivatives and evaluate at (1,2).∂x1∂f=4x1+4x2=12,∂x2∂f=4x1+8x2=20 Answer: ∇f(1,2)=[1220]. The one-line matrix rule matches the term-by-term partial derivatives. Why It Matters
Training a neural network relies on gradients of a loss with respect to whole weight matrices, and backpropagation is the matrix chain rule applied layer by layer. In least-squares regression, the gradient of ∥Ax−b∥2 is 2AT(Ax−b). Setting it to zero gives the normal equations ATAx=ATb. Common Mistakes
Mistake: Using the rule ∇(xTAx)=2Ax for every matrix A. Correction: The shortcut 2Ax holds only when A is symmetric. In the example, 2Ax gives (8,22) instead of the correct (12,20). Use (A+AT)x in general. Also check which layout convention your textbook uses, because the other convention gives the transposed result.
Related Terms
- Multivariable Calculus — The partial derivatives this notation bundles into arrays
- Calculus — The core rules of derivatives this notation extends
- Adjugate — Its transpose gives the gradient of the determinant
- Compatible Matrices — Dimension checks needed for every matrix chain rule product
- Column of a Matrix — Each Jacobian column holds partial derivatives for one input variable
- Jacobian — Matrix of first-order partial derivatives of vector functions