Project Workshop: Model Comparison & Tuning
Now that we have a baseline score from our Logistic Regression model, let’s see if we can do better by trying different models and tuning their hyperparameters! This is one of the most important habits to build as a practicing data scientist: your first working model is almost never your best one, and the only way to know whether a fancier algorithm is worth the extra complexity is to measure it fairly against your baseline.
In this workshop we’ll walk through the full “model selection” workflow that professional teams use: pick a shortlist of candidate algorithms, evaluate each one honestly using cross-validation, and then squeeze extra performance out of the most promising candidate by tuning its hyperparameters. By the end, you’ll have a repeatable process you can apply to almost any tabular dataset.
A baseline is a simple, quick-to-train model that gives you a number to beat. Logistic Regression is a popular baseline because it trains fast, is easy to interpret, and often performs surprisingly well. If a complicated model can’t beat a simple baseline, that’s valuable information too — it might mean your features need more work, or that the extra complexity simply isn’t buying you anything on this dataset.
We will test a few different algorithms to see which performs best on our specific dataset. Each of these algorithms makes different assumptions about the data, so it’s normal for them to perform quite differently on the same problem. We’ll try:
There is no universal “best” algorithm — this is sometimes called the “No Free Lunch” principle. The only reliable way to know which model wins on your data is to actually run the comparison.
To ensure our results are robust and we aren’t just getting “lucky” with our train/test split, we will use 5-fold cross-validation to evaluate each model. This means the training data is split into 5 equal chunks (“folds”). The model is trained on 4 of the folds and tested on the 5th, and this process repeats 5 times so that every fold gets a turn as the test set. Averaging the 5 resulting scores gives us a much more stable estimate of how the model will perform on new, unseen data than a single train/test split ever could.
A model that looks great on one lucky split and terrible on another isn’t actually a good model — it’s an unstable one. Cross-validation is how you catch that before it becomes a production problem.
Once you’ve picked a promising algorithm, the next step is hyperparameter tuning. Hyperparameters are the settings you choose before training starts (like how many trees to grow in a forest, or how deep each tree is allowed to go) as opposed to parameters, which the model learns automatically from data. Grid Search is a straightforward tuning strategy: you list out a handful of candidate values for each hyperparameter, and the algorithm exhaustively tries every combination, scoring each one with cross-validation so the comparison stays fair.
It’s worth noticing how naturally this builds on what you already know. Model comparison answers the question “which family of algorithm fits this problem best?” while tuning answers a narrower follow-up question: “given that we’ve chosen this algorithm, which specific settings squeeze the most performance out of it?” Skipping straight to tuning without first comparing model families is a common shortcut that can leave real performance on the table, since even a perfectly tuned weak algorithm may never catch up to a reasonably-configured strong one.
Think of this process like hiring for a specialized role. First you compare different types of candidates — should you look for a generalist, a specialist, or someone with deep domain expertise? That’s model comparison. Once you’ve settled on the type of candidate that best fits the role, you fine-tune the specifics of the offer — salary, responsibilities, start date — to get the best possible fit. Tuning without first picking the right candidate type wastes effort negotiating details for someone who was never right for the job in the first place.
A few mistakes are easy to make here. First, don’t tune your model against the same data you’ll use for your final evaluation — that leaks information and makes your reported score overly optimistic. Second, don’t assume more complex always beats simpler; a well-tuned simple model can beat a poorly-tuned complex one. Finally, remember that a small improvement in cross-validation score isn’t always worth the added training time and reduced interpretability of a more complex model — that tradeoff is a judgment call, not just a number to maximize.
The coding challenge below puts all of this together: you’ll configure a parameter grid, run Grid Search with cross-validation on a Random Forest, and compare the tuned result against your original baseline.
Let’s tune a Random Forest model using Grid Search.
RandomForestClassifier.n_estimators of 50 and 100, and max_depth of 5 and None.GridSearchCV to find the best combination of parameters using 5-fold cross-validation.best_params_ and the best_score_. Does it beat your baseline?