Lesson 15: Evaluating Regression Models

Evaluating Regression Models

How do we know if our regression model is any good? For numbers, we can't just measure "accuracy" (it's almost impossible to predict exactly $312,456.78). Instead, we measure how far off our predictions are from the true values.

Draggable Regression Error Sandbox

Drag the target data points up or down and compare absolute errors against squared error squares.

ŷ = x (Identity Model)

💡 Tip: Click and drag any node up/down to see how it affects evaluation residuals.

Calculated Metrics

R² Score (Coefficient of Determination)

0.85

Excellent Fit! Model captures most variance.

MAE

9.0

Average absolute error

MSE

95.0

Penalizes large errors

RMSE (Square Root of MSE)9.7

Residual Math Highlights

Outlier Impact: Select the Outlier preset. Notice that while MAE is only 12.0, MSE shoots up to 980.0! This is because the square of a large error dominates the sum.
Why MSE Squares: In Squared Error mode, look at the literal squares drawn. The visual area represents the error term (y_i - ŷ_i)².

Common Regression Metrics

Mean Squared Error (MSE): Averages the squared differences between predictions and actuals. Heavily penalizes large errors!
Root Mean Squared Error (RMSE): The square root of MSE. Useful because it's in the same units as our target variable (e.g., dollars).
Mean Absolute Error (MAE): The average absolute difference. Easier to understand natively.
R² Score: Measures how much of the variance in the target is explained by the model. 1.0 is perfect, 0 means it's no better than guessing the average.

Worked Example: Computing the Metrics by Hand

Before relying on a library function, calculate each metric once yourself so you know what the number means. Take four houses with true prices and predictions, in thousands of dollars:

House	Actual	Predicted	Predicted - actual	Squared error
1	200	210	10	100
2	300	280	-20	400
3	400	400	0	0
4	500	470	-30	900

MAE is the mean of the absolute errors: (10 + 20 + 0 + 30) / 4 = 15. On average the model is off by $15,000. MSE is the mean of the squared errors: (100 + 400 + 0 + 900) / 4 = 350, in squared units that are hard to interpret. RMSE is its square root, about 18.7, back in thousands of dollars. Notice that RMSE (18.7) is larger than MAE (15). RMSE is always at least as large as MAE, and the more uneven the errors, the bigger the gap. Here the single $30,000 miss pulls RMSE up, which is the squaring penalty at work.

For R², compare your squared error to that of a baseline that always predicts the mean. The mean of the actual prices is 350, so the baseline errors are -150, -50, 50 and 150, with squares summing to 50,000. Our model's squared errors sum to 1,400. R² = 1 - 1,400 / 50,000 = 0.972: the model explains about 97 percent of the variation in prices.

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

y_true = np.array([200, 300, 400, 500])
y_pred = np.array([210, 280, 400, 470])

mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_true, y_pred)

print(f"MAE={mae:.1f}  MSE={mse:.1f}  RMSE={rmse:.1f}  R2={r2:.3f}")
# MAE=15.0  MSE=350.0  RMSE=18.7  R2=0.972

Choosing Between MAE and RMSE

MAE treats every dollar of error equally, so it is robust to a few extreme mistakes and easy to explain to non-technical readers. RMSE punishes large errors disproportionately, so it is the better choice when big misses are especially costly, for example when under-estimating a delivery time by an hour is much worse than missing by five minutes twelve times. A useful diagnostic is to compute both: if RMSE is much larger than MAE, a few large errors are dominating, and the next step is to inspect those rows.

Reading R² Correctly

R² = 1 means perfect predictions. R² = 0 means the model is no better than predicting the mean.
R² can be negative on test data. If the model is worse than the mean baseline, scikit-learn reports a negative value. That is a sign of serious overfitting or a broken pipeline, not a bug in the metric.
R² is unitless, so it lets you compare models on the same dataset, but it says nothing about whether the error is acceptable in practice. An R² of 0.9 with an RMSE of $200,000 on house prices may still be useless.
On the training set, adding features never lowers R², even useless ones. Judge a model by its score on held-out data.

Residuals: Look at the Errors, Not Only the Averages

A residual is actual - predicted. A single summary number can hide structure, so plot residuals against predictions. In a healthy linear model they scatter randomly around zero with roughly constant spread. A curve suggests the relationship is non-linear and the model is missing a pattern. A funnel that widens with the prediction means the errors grow with the value, which is common for prices and often fixed by modelling the logarithm of the target. A few isolated points far from the rest are outliers worth checking for data-entry mistakes.

residuals = y_true - y_pred
print(residuals)            # [-10  20   0  30]
print(residuals.mean())     # 10.0

The mean residual of a linear regression with an intercept, fitted by ordinary least squares on its training data, is zero. A clearly non-zero mean on test data suggests the model is systematically over- or under-predicting.

Evaluate on the Right Data

All of these metrics must be computed on data the model has not seen. The training error will almost always look better than the test error. If training RMSE is 10 and test RMSE is 40, the model has memorized noise (overfitting). If both are high and similar, the model is too simple (underfitting). The gap between them is often more informative than either number alone.

Common Pitfalls

Swapping the arguments. scikit-learn expects (y_true, y_pred). For MSE the order does not change the value, but for R² it does.
Comparing RMSE across different targets. An RMSE of 20 is excellent for prices in the hundreds of thousands and terrible for exam scores out of 100. Compare against the spread of the target.
Forgetting the square root. MSE is in squared units; report RMSE if you want the original units.
Reporting only training metrics.

Recap

MAE is the average absolute miss; MSE and RMSE square the errors so large misses count more; RMSE is in the target's units.
R² compares your model to always predicting the mean.
Inspect residuals for patterns, and compute every metric on held-out data.

Python Challenge: Grade Your Model

Calculate MSE and R² score for these predictions using scikit-learn.

from sklearn.metrics import mean_squared_error, r2_score
 
 # Actual house prices and our model's predictions (in thousands)
 y_actual = [300, 450, 200, 600, 350]
 y_pred = [310, 440, 210, 580, 360]
 
 # TODO: Calculate MSE
 # mse = ???
 
 # TODO: Calculate R2 Score
 # r2 = ???
 
 # print(f"MSE: {mse}")
 # print(f"R² Score: {r2}")

These metrics help us compare different models and choose the one that makes the smallest errors on our test set.

House

Actual

Predicted

Predicted - actual

Squared error

200

210

100

300

280

-20

400

500

470

-30

900

import numpy as np from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score y_true = np.array([200, 300, 400, 500]) y_pred = np.array([210, 280, 400, 470]) mae = mean_absolute_error(y_true, y_pred) mse = mean_squared_error(y_true, y_pred) rmse = np.sqrt(mse) r2 = r2_score(y_true, y_pred) print(f"MAE={mae:.1f} MSE={mse:.1f} RMSE={rmse:.1f} R2={r2:.3f}") # MAE=15.0 MSE=350.0 RMSE=18.7 R2=0.972

from sklearn.metrics import mean_squared_error, r2_score # Actual house prices and our model's predictions (in thousands) y_actual = [300, 450, 200, 600, 350] y_pred = [310, 440, 210, 580, 360] # TODO: Calculate MSE # mse = ??? # TODO: Calculate R2 Score # r2 = ??? # print(f"MSE: {mse}") # print(f"R² Score: {r2}")