Least Squares Criterion

What it is

The least squares criterion is a method for finding the function (most commonly a straight line) that best fits a set of data by minimizing the sum of the squared differences between observed values and the values predicted by the function. These squared differences are called residuals. The resulting fitted function is often referred to as the regression line or least squares regression.

How it works

1. Plot the data: independent (predictor) variables on the x-axis and the dependent (response) variable on the y-axis.
2. Specify a model form (e.g., straight line, polynomial, logarithmic, Gaussian).
3. For each data point, compute the residual: observed value minus model prediction.
4. Compute the sum of squared residuals (SSR) and choose model parameters that minimize SSR.

For the linear case (ordinary least squares, OLS), this minimization has a closed-form solution and yields the familiar normal equations; under standard assumptions the OLS estimator is the best linear unbiased estimator (Gauss–Markov theorem). When errors are assumed normally distributed, OLS is also the maximum-likelihood estimator.

Common variants

• Ordinary (linear) least squares — fits a straight line; simplest and most widely used.
• Polynomial least squares — fits a polynomial curve.
• Nonlinear least squares — fits models where the relationship is nonlinear in parameters.
• Transformed models — e.g., logarithmic transforms to linearize relationships.
• Gaussian (and other) basis-function methods — fit sums of basis functions such as Gaussians.

What it tells you

• The fitted model summarizes the relationship between predictors and response variables.
• It provides point estimates of model parameters and predictions for unobserved inputs.
• Residuals and goodness-of-fit measures (e.g., R², root-mean-square error) indicate how well the model captures the data.
• Under appropriate assumptions, parameter estimates have known sampling distributions for inference (confidence intervals, hypothesis tests).

Applications

Least squares methods are widely used across fields:
* Finance and investing — return modeling, risk analysis, option modeling, portfolio simulations.
* Economics — demand/supply estimation, forecasting, policy analysis.
* Engineering and sciences — curve fitting, calibration, signal processing.
* Machine learning — linear regression as a basic supervised learning method.

Limitations and cautions

• Sensitive to outliers: squaring residuals gives large influence to extreme errors.
• Model mis-specification: choosing the wrong functional form yields biased or misleading results.
• Assumptions matter: independence, homoscedasticity (constant variance), and linearity (for OLS) affect validity of inference.
• Overfitting: flexible models (high-degree polynomials) can fit noise instead of underlying relationships.

Key takeaways

• The least squares criterion finds model parameters that minimize the sum of squared residuals, producing a best-fit function for given data.
• Ordinary least squares is the simplest and most common approach; many extensions handle nonlinearity and complex basis functions.
• Proper model selection, diagnostics, and awareness of assumptions are essential for reliable use and interpretation.