Regression analysis
Four data sets can have the same line, the same R² and the same mean, and look nothing alike. That is the Anscombe quartet, and it is the reason a residual plot is not optional.
01The line of best fit
Measuring a sensor’s output at several known temperatures gives a cloud of points, not a line. Regression looks for the line that best represents them, and “best” has a precise definition: the one that minimizes the sum of the squares of the vertical distances.
In electronics this is calibration: you apply known values, record what the instrument reads and fit the line that converts one into the other. The intercept is the offset and the slope is the gain, which are exactly the two parameters that a datasheet specifies.
02How well it fits
It is the most widespread mistake in all of applied statistics. An \( R^2 \) of 0.95 can come from a relationship that is not linear, from a single point that drags the line, or from a cloud with no structure plus one distant value. The Anscombe quartet, which is in the lab below, consists of four data sets with the same mean, the same standard deviation, the same line and the same \( R^2 \), and with shapes that look nothing alike. It is the same argument as the histogram in the first unit: you have to look at the data.
Go through the four Anscombe sets and compare their statistics: they are identical. The residual plot below is what gives them away, and it is the first thing to look at in any fit.
03Residuals are the diagnostic
The residual of each point is what the line did not explain: \( e_i = y_i - \hat{y}_i \). Plotting them against \( x \) is the most useful diagnostic tool there is, because if the linear model is correct no structure should remain in them: just even noise around zero.
| The residuals show | It means | What to do |
|---|---|---|
| A shapeless cloud around zero | The linear model works well | Nothing |
| A curve | The relationship is not linear | Add a quadratic term or transform |
| A funnel that opens up | The spread grows with \( x \) | Weight the fit, or work with the logarithm |
| A very distant point | Outlier or influential point | Investigate it before deciding |
Two different things, and both are common. Two variables being correlated does not mean one causes the other: there may be a third behind them, or pure coincidence. And a line fitted between 0 and 100 °C says nothing about what happens at 300 °C: outside the measured range, the model has no backing from any data. Extrapolating a calibration is one of the most common ways of introducing a large error with a lot of confidence.
04In the lab
Measure the output of a signal-conditioned thermistor at eight known temperatures. Fit the line, obtain the gain and offset, and calculate \( R^2 \). Plot the residuals and decide whether the linear model is sufficient.
Repeat the fit from the previous activity adding a point measured with a gross error. Compare slope, intercept and \( R^2 \) with and without it, and evaluate how far a single data point moved it.
Take the nonlinear curve of a thermistor and plot \( \ln R \) against \( 1/T \). Fit the line on the transformed variables, obtain the constant \( B \) and verify that the residuals now have no structure.
05Common mistakes
- Trusting a high \( R^2 \) without looking at the data. The Anscombe quartet exists for that.
- Not plotting the residuals. It is the cheapest and most informative diagnostic.
- Extrapolating outside the measured range. There the model has no data to support it.
- Reading causation into a correlation. There may be a third variable behind it.
- Forgetting that the line of \( y \) on \( x \) is not that of \( x \) on \( y \). Vertical distances are minimized.
- Adding terms until \( R^2 \) goes up. With enough parameters you can fit anything, including the noise.
06Self-assessment
What does the least-squares method minimize?
The sum of the squares of the vertical distances between the points and the line.
What does \( R^2 \) represent?
The fraction of the variability of \( y \) that the model explains.
What is the Anscombe quartet?
Four data sets with the same line, the same \( R^2 \) and the same mean, and completely different shapes. The proof that you have to plot the data.
What should a correct residual plot show?
Nothing: even noise around zero, with no structure.
What are the gain and the offset of a calibration?
The slope and the intercept of the fitted line.
Why should you not extrapolate?
Because outside the measured range no data support the model, and the error can be large without anything warning you.
07Further reading
- Douglas C. Montgomery and George C. Runger. Applied Statistics and Probability for Engineers. 7th ed. (Spanish edition, “Probabilidad y estadística aplicadas a la ingeniería”), Wiley, 2018. Chapters 11 and 12: simple and multiple linear regression, with inference on the coefficients.
- Francis J. Anscombe. Graphs in Statistical Analysis. The American Statistician, vol. 27, 1973. Four pages in which the quartet first appeared. It is still the best argument in favor of plotting.
- John Fox. Applied Regression Analysis and Generalized Linear Models. 3rd ed., SAGE, 2015. Residual diagnostics, influential points and what to do when the linear model is not enough.