The ML Workflow: From Data to Predictions
Machine Learning isn't just about training a model; it's a complete pipeline from understanding the problem to making predictions. The typical ML workflow involves 7 steps:
Imagine giving a student the answer key to a test before they take it. They would score 100%, but you wouldn't know if they actually learned the material. In ML, we never evaluate our model on the same data it learned from. We split our dataset into a Training Set and a Test Set.
Can you use scikit-learn to split your data into 80% training and 20% testing?
from sklearn.model_selection import train_test_split
import numpy as np
# Let's say X is our features and y is our labels
X = np.arange(100).reshape((50, 2))
y = np.arange(50)
# TODO: Split the data into X_train, X_test, y_train, y_test
# Set test_size to 20%
# X_train, X_test, y_train, y_test = ???
print(f"Training samples: {len(X_train)}")
print(f"Testing samples: {len(X_test)}")train_test_split shuffles the rows and then divides them. Passing random_state fixes the shuffle so your results are repeatable. For classification, passing stratify=y keeps the share of each class the same in both parts, which matters when one class is rare. With 50 rows and test_size=0.2 you get 40 training rows and 10 test rows.
import numpy as np
from sklearn.model_selection import train_test_split
X = np.random.rand(100, 3)
y = np.array([0] * 90 + [1] * 10) # imbalanced: 10% positives
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=0
)
print(len(X_tr), len(X_te)) # 80 20
print(y_tr.mean(), y_te.mean()) # 0.1 0.1 (same class ratio)Leakage happens when information from the test set influences training. The most common cause is preprocessing the whole dataset before splitting. If you compute the mean and standard deviation for scaling on all rows, the training step has already "seen" a summary of the test rows. Fit every preprocessing step on the training data only, then apply it to the test data. A scikit-learn Pipeline does this for you automatically:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_tr, y_tr) # scaler is fitted on training data only
print(pipe.score(X_te, y_te)) # test data is only transformed, never fittedAccuracy is the share of correct predictions, and it can be misleading. In the imbalanced example above, a model that always predicts 0 is 90% accurate and finds none of the positives. For such problems look at precision (of the items flagged positive, how many truly were) and recall (of the truly positive items, how many were flagged). For regression, common choices are mean absolute error and R-squared.
When training, a model might memorize the training data (Overfitting) or fail to capture the pattern at all (Underfitting). We want to find the sweet spot in the middle!
The signature of overfitting is a large gap between training and test performance: near-perfect on data the model has seen, noticeably worse on data it has not. Underfitting shows up as poor scores on both. A decision tree makes the effect easy to see, because its max_depth setting controls how flexible it is:
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = make_moons(n_samples=300, noise=0.35, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
for depth in [1, 3, None]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)
print(depth, round(tree.score(X_tr, y_tr), 2), round(tree.score(X_te, y_te), 2))With max_depth=None the tree keeps splitting until it classifies every training point correctly, so its training score is 1.00, but on noisy data like this its test score is typically lower than that of a shallower tree. A depth of 1 is too simple to follow the curved boundary and scores modestly on both sets. Run it and compare the three rows; the exact numbers depend on the data and the split.
If you pick settings such as max_depth by checking the test score, the test set quietly becomes part of training. Instead, use cross-validation on the training data to compare settings, and only look at the test set once, at the end:
from sklearn.model_selection import cross_val_score
for depth in [1, 3, 5, None]:
scores = cross_val_score(DecisionTreeClassifier(max_depth=depth, random_state=0), X_tr, y_tr, cv=5)
print(depth, scores.mean().round(3))The workflow is a loop, not a line: define the problem, explore and prepare the data, split before any fitting, train, evaluate on held-out data with a metric that fits the problem, and iterate using cross-validation. Whenever a score looks too good, check for leakage first.