Mathwords logoMathwords

Least Squares Fitting — Definition, Formula & Examples

Least squares fitting is a method for finding the line (or curve) that best fits a set of data points by minimizing the total of the squared differences between observed values and predicted values.

Given nn data points (xi,yi)(x_i, y_i), least squares fitting determines the parameters of a model function y^=f(x;β)\hat{y} = f(x; \boldsymbol{\beta}) by minimizing the objective S=∑i=1n(yi−y^i)2S = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2, where y^i\hat{y}_i is the value predicted by the model for observation ii.

Key Formula

S=∑i=1n(yi−y^i)2S = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2
Where:
  • SS = Sum of squared residuals (the quantity to minimize)
  • yiy_i = Observed value of the response variable for data point i
  • y^i\hat{y}_i = Predicted value from the fitted model for data point i
  • nn = Number of data points

How It Works

For each data point, you compute the residual — the vertical distance between the observed yy value and the value your model predicts. Squaring each residual ensures that positive and negative errors don't cancel and that larger errors are penalized more heavily. The fitting procedure then adjusts the model's parameters (slope and intercept for a line) until the sum of those squared residuals is as small as possible. In the case of simple linear regression, closed-form formulas exist for the optimal slope bb and intercept aa, so no iterative searching is needed.

Worked Example

Problem: Fit a least squares line to the data points (1, 2), (2, 4), (3, 5), (4, 4), (5, 5).
Step 1: Compute the means of x and y.
xˉ=1+2+3+4+55=3,yˉ=2+4+5+4+55=4\bar{x} = \frac{1+2+3+4+5}{5} = 3, \quad \bar{y} = \frac{2+4+5+4+5}{5} = 4
Step 2: Compute the slope using the least squares formula.
b=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2=(−2)(−2)+(−1)(0)+(0)(1)+(1)(0)+(2)(1)4+1+0+1+4=610=0.6b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)}{4+1+0+1+4} = \frac{6}{10} = 0.6
Step 3: Compute the intercept.
a=yˉ−bxˉ=4−0.6(3)=2.2a = \bar{y} - b\bar{x} = 4 - 0.6(3) = 2.2
Answer: The least squares regression line is y^=0.6x+2.2\hat{y} = 0.6x + 2.2.

Why It Matters

Least squares fitting is the foundation of linear regression, which appears in nearly every statistics, econometrics, and data science course. Scientists and engineers use it to calibrate instruments, model trends, and make predictions from experimental data.

Common Mistakes

Mistake: Minimizing the sum of residuals instead of the sum of squared residuals.
Correction: Positive and negative residuals cancel out when summed directly, so the raw sum can be zero even for a terrible fit. Squaring the residuals prevents this cancellation and properly penalizes all deviations.