Lesson 27: Project Workshop — Feature Engineering & Baseline

Project Workshop: Feature Engineering & Baseline Model

With our data explored, it's time to prepare it for our machine learning algorithms. We need to convert our raw text into numbers and establish a baseline model.

Step 1: Text Preprocessing and Feature Engineering

Our models cannot read English; they need numbers. We will use a TfidfVectorizer to convert the text into TF-IDF features.

Step 2: Train/Test Split

We must set aside some data to evaluate our model later. A common split is 80% for training and 20% for testing. Make sure to use stratified sampling so both sets have an equal balance of positive and negative reviews.

Step 3: Baseline Model

A baseline model is a simple, standard model we use as a benchmark. If a complex model can't beat the baseline, it's not worth using! We will use Logistic Regression.

What TF-IDF Actually Computes

A plain bag-of-words count treats every word equally, so common words such as "the" or "movie" dominate the counts even though they say little about sentiment. TF-IDF fixes this by multiplying two quantities. The term frequency (TF) is how often a word appears in this review. The inverse document frequency(IDF) is larger for words that appear in few reviews and smaller for words that appear in almost all of them. A word like "masterpiece" that appears in a small share of reviews gets a high weight, while "movie" gets a low one. Scikit-learn then rescales each review's vector to unit length so long and short reviews are comparable. The result is a sparse matrix with one row per review and one column per vocabulary word.

Worked Example: Split First, Then Fit

The order of operations matters. The vectorizer learns its vocabulary and IDF weights from whatever you call fit on, so it must only ever see the training reviews. The code below assumes your DataFrame has columns named review and sentiment; adjust the names to match your data.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(
    df["review"], df["sentiment"],
    test_size=0.2, stratify=df["sentiment"], random_state=42,
)

vec = TfidfVectorizer(max_features=3000, stop_words="english")
X_train_tfidf = vec.fit_transform(X_train)   # learn vocabulary + IDF from train only
X_test_tfidf = vec.transform(X_test)         # reuse them on the test set

print(X_train_tfidf.shape)   # (number of training reviews, up to 3000)

model = LogisticRegression(max_iter=1000)
model.fit(X_train_tfidf, y_train)
print(accuracy_score(y_test, model.predict(X_test_tfidf)))

With 2,000 reviews and an 80/20 split you get 1,600 training and 400 test reviews, and stratification keeps the positive/negative ratio the same in both. Setting random_state makes the split repeatable, so a score you report today can be reproduced tomorrow.

Sanity Checks Before You Trust the Score

Two quick checks make your baseline meaningful. First, compare against a model that ignores the text entirely. Second, look inside the vocabulary to confirm the features are sensible.

from sklearn.dummy import DummyClassifier

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train_tfidf, y_train)
print("majority-class accuracy:", dummy.score(X_test_tfidf, y_test))

print(vec.get_feature_names_out()[:20])   # a peek at the learned vocabulary

On a balanced dataset the dummy model scores about 50%, which is the floor your real baseline has to clear. If your Logistic Regression result is only a point or two above it, something is wrong with the features or the labels.

Common Pitfalls

Data leakage: calling fit_transform on the full dataset before splitting lets test reviews influence the vocabulary and IDF values. The score looks slightly better than it should. Always split first.
Calling fit on the test set: use transform only. Refitting would give the test data a different set of columns than the model was trained on.
Forgetting stratify: on a small dataset an unlucky random split can skew the class ratio and make the accuracy noisy.
Judging by one number: with 400 test reviews, one percentage point is only four reviews. Treat tiny differences between models as noise until cross-validation says otherwise.

Recap

Split the data first, fit the vectorizer on the training text only, transform both sets, and train a simple Logistic Regression as your benchmark. Check it against a majority-class dummy and write down the score: every model in the next lesson has to beat it to justify its extra complexity.

Coding Challenge: Build the Baseline

Put together your preprocessing and baseline model:

Fit a TfidfVectorizer on the training data. Let's try max_features=3000.
Transform both the training and test data.
Train a LogisticRegression model on the training data.
Predict on the test data and calculate the accuracy. This is your baseline score!

from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score X_train, X_test, y_train, y_test = train_test_split( df["review"], df["sentiment"], test_size=0.2, stratify=df["sentiment"], random_state=42, ) vec = TfidfVectorizer(max_features=3000, stop_words="english") X_train_tfidf = vec.fit_transform(X_train) # learn vocabulary + IDF from train only X_test_tfidf = vec.transform(X_test) # reuse them on the test set print(X_train_tfidf.shape) # (number of training reviews, up to 3000) model = LogisticRegression(max_iter=1000) model.fit(X_train_tfidf, y_train) print(accuracy_score(y_test, model.predict(X_test_tfidf)))

from sklearn.dummy import DummyClassifier dummy = DummyClassifier(strategy="most_frequent") dummy.fit(X_train_tfidf, y_train) print("majority-class accuracy:", dummy.score(X_test_tfidf, y_test)) print(vec.get_feature_names_out()[:20]) # a peek at the learned vocabulary