Lesson 23: Text Data & NLP Basics

Welcome to Natural Language Processing (NLP)!

The challenge with text is that computers only understand numbers, not words. To apply Machine Learning to text, we must first convert the text into numbers.

The Text Preprocessing Pipeline

Before converting text to numbers, we usually clean it up. The standard pipeline looks like this:

Lowercasing: Making everything lowercase so "Apple" and "apple" are treated the same.
Removing Punctuation: Stripping out commas, periods, etc.
Tokenization: Splitting a sentence into a list of individual words (tokens).
Removing Stop Words: Dropping common words like "the", "is", and "a" that don't add much meaning.

Bag of Words (BoW) & TF-IDF

Once text is tokenized, we represent it using numbers. The simplest way is a Bag of Words, where we just count the frequency of each word.

A smarter approach is TF-IDF (Term Frequency - Inverse Document Frequency). It gives more weight to words that appear frequently in a specific document but rarely across all documents. For example, in a movie review, "cinematography" is much more informative than "movie".

Why Order Is Lost, and Why That Is Often Fine

A bag of words discards word order completely: "dog bites man" and "man bites dog" produce the same counts. For many tasks such as topic classification or spam filtering this loss barely matters, because the presence of certain words carries most of the signal. For tasks that depend on negation, such as telling "not good" from "good", you can keep short phrases by using n-grams, where bigrams treat "not good" as one feature.

Worked Example: Building a Bag of Words by Hand

Take two tiny documents. First build the vocabulary (the list of distinct words), then count each word per document.

docs = ["the movie was great", "the movie was boring boring"]

vocab = sorted({w for d in docs for w in d.split()})
print(vocab)   # ['boring', 'great', 'movie', 'the', 'was']

for d in docs:
    words = d.split()
    print([words.count(v) for v in vocab])
# [0, 1, 1, 1, 1]
# [2, 0, 1, 1, 1]

Each document is now a vector of length 5 and can be fed to any numeric model. In real corpora the vocabulary has tens of thousands of words, so these vectors are long and mostly zeros; scikit-learn stores them as sparse matrices to save memory.

The Same Thing with scikit-learn

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

docs = ["the movie was great", "the movie was boring boring"]

cv = CountVectorizer()
X = cv.fit_transform(docs)
print(cv.get_feature_names_out())   # ['boring' 'great' 'movie' 'the' 'was']
print(X.toarray())

tfidf = TfidfVectorizer()
print(tfidf.fit_transform(docs).toarray().round(2))

fit learns the vocabulary from the training texts, and transform converts texts using that fixed vocabulary. Words that appear only at prediction time and were never seen in training are simply ignored.

How TF-IDF Weights Words

TF-IDF multiplies two quantities. Term frequency is how often a word occurs in this document. Inverse document frequency is roughly log(N / df), where N is the number of documents and dfis how many contain the word, so words that appear everywhere get a weight near zero while distinctive words get a large one. In the example above, "the", "movie" and "was" occur in both documents and are down-weighted, while "great" and "boring" stand out. (scikit-learn adds smoothing and normalizes each row to unit length, so its numbers differ slightly from the textbook formula.)

Preprocessing Choices Are Not Free

Stop-word removal shrinks the vocabulary, but it can delete meaning: "not" is on many stop lists, and dropping it flips the sense of "not good".
Punctuation can carry signal. An exclamation mark may matter for sentiment.
Stemming and lemmatization merge "running", "runs" and "ran" into one form, which reduces sparsity at the cost of nuance.
Case: lowercasing merges "Apple" the company with "apple" the fruit.

Treat each cleaning step as a hypothesis and measure it on validation data rather than applying a fixed recipe.

Common Pitfalls

Fitting the vectorizer on all the data before splitting. Fit on the training texts only, then transform the test texts.
Splitting on spaces only. "fantastic!".split() keeps the exclamation mark attached to the word.
Assuming a huge vocabulary helps. Rare words add noise; min_df and max_features cap the size.

Recap

Models need numbers, so text is cleaned, tokenized and turned into vectors.
Bag of words counts tokens and ignores order; n-grams recover some local order.
TF-IDF down-weights words that appear in many documents.
Learn the vocabulary from training data only.

Coding Challenge: Tokenization

Let's practice some basic string manipulation in Python to tokenize a sentence.

Create a variable review = "The movie was absolutely fantastic!".
Convert the review to lowercase using review.lower() and store it.
Remove the exclamation mark by using .replace("!", "").
Split the string into a list of words using .split() and print the list.

docs = ["the movie was great", "the movie was boring boring"] vocab = sorted({w for d in docs for w in d.split()}) print(vocab) # ['boring', 'great', 'movie', 'the', 'was'] for d in docs: words = d.split() print([words.count(v) for v in vocab]) # [0, 1, 1, 1, 1] # [2, 0, 1, 1, 1]

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer docs = ["the movie was great", "the movie was boring boring"] cv = CountVectorizer() X = cv.fit_transform(docs) print(cv.get_feature_names_out()) # ['boring' 'great' 'movie' 'the' 'was'] print(X.toarray()) tfidf = TfidfVectorizer() print(tfidf.fit_transform(docs).toarray().round(2))