Train, Validate, Test: Doing It Right
We've talked about a Train/Test split. But there is a subtle problem: if you train a model, test it, tweak the settings to get a better test score, and test again... you are accidentally "leaking" information from the test set into the model! You're optimizing for the test set.
The gold standard is a 3-way split: Train (to learn the patterns), Validate (to tune the model's settings), and Test (held in a locked vault until the very end for a final grade).
What if your validation set just happens to be unusually easy or hard? Enter K-Fold Cross-Validation. We chop our training data into K chunks (folds). We train on K-1 chunks and validate on the 1 remaining chunk. We repeat this K times, rotating the validation chunk, and average the scores. This gives us a much more robust estimate of how the model will perform.
Every time you look at a test score and change something in response, you transfer a little information about the test set into your decisions. After dozens of tweaks the model is partly fitted to that specific test sample, and its reported score becomes optimistic. The same effect, at a larger scale, is why results on public benchmarks tend to be higher than real-world performance. The remedy is structural: tune against one dataset and grade against another that you have not touched.
With scikit-learn the easiest approach is to call train_test_split twice. First set aside the test set, then split what remains into train and validation. A common ratio is 60 / 20 / 20.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
X_temp, X_test, y_temp, y_test = train_test_split(
X, y, test_size=0.20, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)
print(len(X_train), len(X_val), len(X_test)) # 90 30 30The second split uses test_size=0.25 because 25 percent of the remaining 80 percent is 20 percent of the whole. The random_state makes the shuffle reproducible, and stratify=y keeps the class proportions equal in every part, which matters when some classes are rare.
In 5-fold cross-validation each sample is used for validation exactly once. You get five scores, and both their mean and their spread are informative. A mean of 0.95 with scores between 0.93 and 0.97 describes a stable model. A mean of 0.95 with scores between 0.80 and 1.00 tells you the result depends heavily on which samples land in which fold, so you should not trust the average blindly. Report the mean together with the standard deviation:
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=50, random_state=0)
scores = cross_val_score(model, X, y, cv=5)
print(f"{scores.mean():.3f} +/- {scores.std():.3f}")cross_val_score clones and refits the model for every fold, so it never reuses a model that has already seen the validation data. For classifiers, passing an integer cv uses stratified folds automatically.
Data leakage means information from outside the training data sneaks into training, producing scores that vanish in production. The classic mistake is preprocessing before splitting. If you scale all the data with its overall mean and standard deviation and only then split, the training set has absorbed statistics from the validation rows. The correct pattern is to fit any transformer on the training portion only. In scikit-learn a Pipeline does this for you inside cross-validation:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
print(cross_val_score(pipe, X, y, cv=5).mean())Other leakage sources include features that are only known after the outcome (for example a "refund issued" flag when predicting fraud), and duplicate rows that appear in both train and test sets. For time-ordered data, random shuffling leaks the future into the past; use a time-based split instead, training on earlier dates and validating on later ones.
Run a 5-fold cross validation on a Random Forest.
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
model = RandomForestClassifier(n_estimators=10)
# TODO: Use cross_val_score with cv=5 to get 5 accuracy scores
# scores = ???
# print(f"Scores for each fold: {scores}")
# print(f"Average accuracy: {scores.mean():.2f}")