Project Workshop: Define the Problem and Explore the Data
It’s time for the capstone project! You will build a complete sentiment analysis system that classifies movie reviews as positive or negative. We’ll put together everything you’ve learned: data exploration, preprocessing, feature engineering, model selection, evaluation, and interpretation. Unlike the shorter exercises earlier in the course, this project spans multiple lessons and mirrors how a real machine learning project actually unfolds in the wild — messy, iterative, and rarely a straight line from raw data to a finished model.
Sentiment analysis is one of the most widely used applications of machine learning in industry. Companies use it to automatically gauge customer opinion from product reviews, social media posts, and support tickets at a scale no human team could read manually. The techniques you practice here — turning text into numeric features, training a classifier, and evaluating it honestly — transfer directly to email spam detection, content moderation, and countless other text classification problems you’ll encounter in the field.
We are dealing with a binary text classification problem. Given the text of a movie review, our goal is to predict the label: Positive or Negative. Framing the problem clearly before writing a single line of code is a habit worth building early. Ask yourself: what exactly is the input, what exactly is the output, and how will you know if a prediction is “good enough” for the intended use case? For this project, the input is raw review text and the output is a single binary label — deliberately simple, so we can focus on the full pipeline without getting lost in an overly complex target.
We have a dataset of 2,000 movie reviews (1,000 positive and 1,000 negative). Before jumping into modeling, you must understand your data. This step is often called Exploratory Data Analysis (EDA), and skipping it is one of the most common mistakes beginners make. Jumping straight to training a model on data you don’t understand is a bit like trying to cook a meal from a box of unlabeled ingredients — you might get lucky, but you won’t know why something went wrong if it doesn’t turn out well.
A few questions are worth answering during exploration: Are the classes balanced, or is one label far more common than the other? What does a typical review actually look like — is it a full paragraph, a single sentence, or something in between? Are there quirks in the text, like HTML tags, punctuation, or unusual capitalization, that you’ll need to clean up before feeding the data into a model? Answering these questions now will save you from confusing bugs and misleading results later in the project.
Class balance deserves special attention because it directly affects how you should evaluate your eventual model. If a dataset were, say, 95% positive and only 5% negative, a lazy model that always predicts “Positive” would score 95% accuracy — while being completely useless at its actual job of identifying negative reviews. Confirming the balance of your dataset upfront tells you whether plain accuracy will be a trustworthy metric later, or whether you’ll need to lean on metrics like precision, recall, or a confusion matrix instead.
Once exploration is complete, the next lessons in this workshop series will walk through cleaning the review text (removing noise like HTML tags and punctuation), converting text into numeric features a model can understand (using techniques like bag-of-words or TF-IDF), training a classifier, and finally evaluating its performance with metrics beyond plain accuracy. Each step builds directly on the exploration you do here — the cleaning decisions you make depend on the quirks you discover in this step, and the evaluation metrics you choose depend on the class balance you confirm here.
The coding challenge below walks you through exactly this kind of exploration using pandas, one of the standard tools for working with tabular and text data in Python.
Use pandas to explore the dataset and answer these questions:
df['sentiment'].value_counts(). Are the classes perfectly balanced?It might feel like exploration is “just looking at the data” rather than “real machine learning,” but the choices you make here quietly shape everything that follows. If you discover the classes aren’t perfectly balanced, you’ll know to track precision and recall alongside accuracy later on. If you discover that negative reviews tend to be noticeably longer than positive ones, that’s a hint that review length itself might be a useful engineered feature down the line. Experienced practitioners spend a meaningful fraction of any real project just getting to know their data — not because it’s glamorous, but because it consistently pays off in fewer surprises and better-informed decisions later.
Keep careful notes as you explore. When you move on to preprocessing and modeling in the next parts of this workshop, you’ll want to refer back to what you found here — the class balance, the typical review length, and any quirks in the raw text — to justify the decisions you make at each subsequent step.